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(54) Image processing apparatus 



(57) In an apparatus and method lor creating a 
three-dimensional model of an object, images of the ob- 
ject taken from different, unknown positions are proc- 
essed to identify the points in the images which corre- 
spond to the same point on (he actual object (that is 
"matching" points),. the matching points are used to de- 
termine the relative positions from which the images 
were taken, and the matching points and calculated po- 
sitions are used to calculate points in a three-dimension- 
al space representing points on the object. A number ol 
different techniques are used to identify the matching 
points, and a number of solutions are calculated and 
tested for the relative positions, the solution which is 
consistent with Ihe largest number ol matching points 
being selected, in one matching technique, edges in an 
image are identified by first identifying comer points in 
the image and then identifyrig edges between the cor- 
ner points on the basis of edge orientation values of pix- 
els, the edges are processed hn strength order to remove 
cross-overs, the images sub-divided into regions by 
connecting points at the ends of the edges on the basis 
of the edge strengths, and matching points within cor- 
responding regions in I wo or more images are identified. 
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Description 

[0001] The present invention relates to an image processing apparatus and method. 

[0002] Creating three-dimensional computer models of a real-life object has traditionally been time consuming and 
s expensive, requiring skilled personnel and/or expensive equipment One reason for this is that ft is necessary to de- 
termine the relative 'depth' of points on the object as well as the relative positions of the points in two dimensions. 
[0003] Known techniques for creating three-dimensional computer models from real objects fall into one of two cat- 
egories, namely active techniques in which depth information is obtained by actively sensing the surface of the object, 
and passive techniques in which depth information is obtained from images of the object. 
10 [0004] Examples of active techniques include scanning the object with a pulsed laser beam and measuring the 
detection time of pulses relative to their transmission time to determine depth information (as in a laser 'rangefmder") : 
and touching the object at a number of points on its surface with a position-sensitive probe. 
[0005] In conventional passive techniques, at least two images of the object taken from different camera positions 
are needed. To construct a three-dimensional computer model from the images, it Is necessary to know firstly the 
is location in each image of points which represent the same actual point on the object, and secondly the relative positions 
from which the images were taken. These are particularly onerous requirements. As a result, in known passive systems, 
distinguishing marks/calibrations are added to the object or its surroundings to enable matching points to be easily 
identified in the images, and/or the images are taken from known camera positions. 

[0006] For example, WO-A-90/10194 discloses a system lor measuring strain distribution in an object using the 
20 three-dimensional coordinates of points on the object surface calculated from two images of the object, "fo facilitate 

the matching of points in the images, a uniform square grid pattern is applied to the object before the images are taken, 

by electrochemical etching or silk screening. Corresponding points of intersection of the grid lines can then be easily 

identified in the images. Further, the object is placed on a rotary table which is rotaled by a known angle between 

images, thereby defining the relative camera positions. 
2S [0007] US 4803645 discloses a system for determining three-dimensional coordinates of points on an object in which 

a grid or similar periodic pattern is projected onto the object using a light projector, and images of the object are taken 

using three imaging systems at fixed, predefined positions. 

[0008] WO-A-88/02518 discloses a system for producing a depth map of an object from a plurality of images taken 
from imaging devices set in a predefined, known configuration. Similarly, US 5307136 discloses an automobile distance 
30 detection system which determines the range of an automobile using images taken from a plurality of cameras mounted 
in a predefined, known configuration in the user's automobile. 

[0009] GB-A-2244621. WO-A-92/06444 and US 4695156 all disclose systems for determining three<Jimensional 

coordinates of points on an object surface from stereo images of the object taken at known camera positions. 

[001 0] In many cases, however, it is inconvenient, expensive, and/or inf easible to provide a reference grid or mark- 

35 ings/calibrations on the object or its surroundings, or to take images from known relative positions. Reliable and ac- 
curate techniques are then required to match points in the mages and/or to calculate the relative camera positions. 
[0011] Even if matching points are identified using reference features added to the object or its surroundings, an 
accurate and reliable technique for determining the positions at which the images were taken is necessary if the images 
were not taken from known relative positions. 

40 [001 2] Whether or not reference mar\3?gs on the object or its surroundings and/or known camera positions are used, 
when a passive technique is employed, a method which accurately generates three-dimensional coordinate points 
using the matching image points and the camera positions is needed. This is particularly important, however, when no 
relerence markings are used and the camera positions are not known, since inaccuracies in the matching points iden- 
tified in the images and inaccuracies in the calculated camera positions can cause inaccuracies in the calculated three- 

4£ dimensional coordinates of points. 

[001 3] The present invention aims lo address one or more of the above problems. 

[0014] In one aspect, the present invention aims to provide an apparatus and method for determining the positions 
at which images of an object were taken. 

[0015] In another aspect, the present invention aims to provide an apparatus and method for determining points in 
so a three-dimensional space, using matching image features and the relative positions of the images. 

[001 6] The present invention provides an image processing apparatus or method in which the positional relationship 
between at least three images is determined a plurality of times using matching features in the images, each time on 
the basis of a different combination of transformations between the images, with the most accurate relationship being 
selected. Preferably, the different combinations of affins and perspective transformations between the images are 
55 considered. The transformation between one or more pairs of the images may be fixed. 

[001 7] The present invention provides an image processing apparatus or method in which signals defining features 
matched using different techniques in first and second images are processed without using prior information on the 
relationship between the positions from which the images were taken to determine the relationship. The relationship 
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is determined a plurality ol times, each time using features matched with a different matching technique or techniques, 
to give a selection of relationships. 

[0018] The present invention provides an image processing apparatus or method in which signals defining features 
matched in first and second images are processed without using prior information on the relationship between the 
5 positions from which the Images were taken to determine the relationship. The following steps are performed a plurality 
of times: matched features are used to set up a non-physically realisable matrix (such as the fundamental matrix, lor 
example), which is then converted into a physically realisable matrix (such as the physical fundamental matrix, tor 
example) and its accuracy is tested. The most accurate phyeicaJly realisable matrix is selected. 
[0019] The present invention provides an image processing apparatus or method h which signals defining corre- 
io sponding object features in at least three images of the object and signals defining the relationships between the 
imaging positions are processed to produce points in a three-dimensional space representing points on the object. 
Each parr ol corresponding features ts used to define a point in the 3D space, and the 3D points produced from the 
same corresponding features are used to define a 3D point representing a point on the object Processing is then 
performed to see if the 3D object points could represent the same point on the object, and. if they could, further process- 
is ing is performed. 

[0020] Embodiments of the invention wiH now be described by way of example only with reference to the accompa- 
nying drawings, in which: 

[0021] Figure 1 schematically shows the components of an image processing apparatus in an embodiment of the 
invention. 

20 [0022] Figure 2 illustrates the collection of image data by imaging an object from different positions around the object. 
[0023] Figure 3 shows, at a top level, the processing operations performed by the image processing apparatus of 
Figure 1 in an embodiment of the invention. 

[0024] Figure 4 shows the steps performed during initial data input at step S2 in Figure 3. 

[0025] Figure 5 illustrates the sequencing of images by a user at step S22 in Figure 4. 
25 [0026] Figure 6 shows the relationship between the operations in Figure 1 of initial feature matching at step S4, 

calculating camera transformations at step S6 and constrained feature matching at step S8. 

[0027] Figure 7 shows in greater detail the relationship between the operations shown in Figure 6. 

[0028] Figure 8 shows the operations performed during automatic initial feature matching across the first pair of 

images in a triple of images at step S52 in Figure 7. 
30 [0029] Figure 9 shows the operations performed during automatic initial feature matching across the second pair of 

images in a triple of images at step S54 in Figure 7. 

[0030] Figure 10a and Figure 10b schematically illustrate a 'perspective' image and an ■affine' image, respectively. 
[0031] Figure 11 shows, at a top level, the operations performed during affine initial feature matching for the first (or 
second) pair of images in a triple of images at step S62 or step S64 in Figure 7. 
35 [0032] Figure 1 2 shows the operations performed in finding the edges in each image of a pair of images at step S100 
in Figure 11. 

[0033] Figure 13 illustrates the pixels which are considered when calculating edge strengths at step S106 or step 
S 108 in Figure 12. 

[0034] Figure 14 shows the operations performed when calculating edge strengths at step S10B and step SI 08 in 
Figure 12. 

[0035] Figure 15 shows the operations performed when removing edges which cross over other edges at step S112 
in Figure 12. 

[0036] Figure 16a, Figure 16b and Figure 16c show examples of two edges, Figures 16a and 16b showing examples 
in which the edges do not cross, and Figure 16c showing an example in which the edges do cross. 
<* [0037] Figure 17 shows the operations performed when triangulating points at step S 102 in Figure 11. 

[0038] Figure 1 8 shows the operations performed when calculating further corresponding points in a pair ol images 
at step S 104 in Figure 11. 

[0039] Figure 19 illustrates the use of a grid ol squares at steps SI 62, S174 and S1B0 in Figure IB. 
[0040] Figure 20 shows, at a top level, the operations performed when calculating the camera transformations lor a 
50 triple of images at steps S56 and S66 in Figure 7. 

[0041] Figure 21 shows, al atop level, the operations performed when carrying out processing routine 1 at step S202 
in Figure 20. 

[0042] Figure 22 shows the operations performed when setting up the parameters at step S20S in Figure 21 . 
[0043] Figure 23 shows the operations performed in determining the number of iterations to be carried out at step 
« S224 in Figure 22. ' 

[0044] Figure 24 shows, at a top level, the operations performed when calculating the camera transformations for a 

first pair of images in a triple or a second pair of images in a triple at step S208 or step S210 in Figure 21. 

[004S] Figure 25 shows the operations performed when carrying out a perspectrve calculation for an image pair at 
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step S240 in Figure 24. 
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[0072] Figure 1 is a block diagram showing the general arrangement of an image processing apparatus in an em- 
bodiment. In ihe apparatus, there is provided a computer 2, which comprises a central processing unit (CPU) 4 con- 
nected to a memory S operable to store a program defining the operations to be performed by the CPU 4, and to store 
object and image data processed by CPU 4. 

s [0073] Coupled to the memory 6 is a disk drive 8 which is operable to accept removable data storage media, such 
as a floppy disk 1 0, and to transfer data stored thereon to the memory 6. Operating instructions for the central processing 
unit 4 may be input to the memory 6 from a removable data storage medium using the disk drive 8. 
[0074] Image data to be processed by the CPU 4 may also be input to the computer 2 from a removable data storage 
medium using the disk drive 6. Alternatively, or in addition, image data to be processed may be input to memory 6 

if directly from a camera 1 2 having a digital image data output, such as the Canon Powershot 600. The image data may 
be stored in camera 1 2 prior to input to memory 6 : or may be transferred to memory 6 in real time asthe data is gathered 
by camera 12. Image data may also be input from a conventional film camera instead of digital camera 12. In this case, 
a scanner (not shown) is used to scan photographs taken by the camera and to produce digital image dala therefrom 
for input to memory 6 In addition, image data may be downloaded into memory 6 via a connection (not shown) from 

is a local database, such as a Kodak Photo CD apparatus in which image data is stored on optical disks, or from a remote 
database which stores the image data. 

[0075] Coupled to an input port of CPU 4, there is an input device 1 4, which may comprise, for example, a keyboard 
and/or a position sensitive input device such as a mouse, a trackerball, etc. 

[0076] Also coupled to the CPU 4 is a frame buffer 16 which comprises a memory unit arranged to store image data 
& relating to at least one image generated by the central processing unit 4, for example by providing one (or several) 
memory location(s) for a pixel of the image. The value stored in the frame buffer for each pixel defines the colour or 
intensity of that pixel in the image. 

[0077] Coupled to the frame buffer 16 is a display unit 18 for displaying the image stored in the frame buffer 16 in a 
conventional manner. Also coupled to the frame bufler 16 is a video tape recorder (V7R) 20 or other image recording 
2$ device, such as a paper printer or 35mm film recorder. 

[0078] A mass storage device, such as a hard disk drive, having a high data storage capacity, is coupled to the 
memory 6 (typically via the CPU 4), and also to the frame bufler 16. The mass storage device 22 can receive data 
processed by the central processing unit 4 from the memory 6 or data from the frame buffer 1 6 which is to be displayed 
on display unit 18. 

30 [0079] The CPU 4, memory 6, frame buffer 16, display unit 18 and Ihe mass storage device 22 may form part ol a 
commercially available complete system, for example a workstation such as the SparcStation available from Sun Mi- 
crosystems. 

[0080] Operating instructions for causing the computer 2 to perform as an embodiment of the invention can be sup* 
plied commercially in the form of programs stored on floppy disk 1 0 or another data storage medium, or can be trans- 
35 mitted as a signal to computer 2. for example over a datalink (not shown), so that the receiving computer 2 becomes 
reconfigured into an apparatus embodying the invention. 

[0081] Figure 2 illustrates the collection of image data for processing by the CPU 4. 

[0082] An object 24 is imaged using camera 12 from a plurality of different locations. By way of example, Figure 2 
illustrates the case where object 24 is imaged Irom five different, random locations labelled L1 to L5 ( with the arrows 
40 in Figure 2 illustrating the movement of the camera 12 between the different locations. 

[0083] Image data recorded at positions L1 to L5 is stored in camera 12 and subsequently downloaded into memory 
6 ol computer 2 for processing by the CPU 4 in a manner which wiil now be described. In this embodiment, CPU 4 
does not receive information defining the locations at which the images were taken, either in absolute terms or relative 
to each other. 

4* [0084] Figure 3 shows the top-level processing routines performed by CPU 4 to process the image data from camera 

[0085] At step S2, a routine for initial data input is performed, which will be described below with reference to Figures 
4 and 5. The aim of this routine is to store Ihe image data received from camera 12 in a manner which facilitates 
subsequent processing, and to store information concerning parameters of the camera 1 2. 
£0 [0086] At step $4, initial feature matching is performed to match features within the different images taken of the 
object 24 (that is, to identify points in the Images which correspond to the same physical point on object 24). This 
process will be described below with reference to Figures 6 to 19. 

[0087] At step S6, the transformations between the different camera positions from which the images were taken 
(LI to L5 in Figure 2), and hence the positions themselves in relative form, are calculated using the points matched in 
ss the images, as win be described below with reference to Figures 20-38. 

[0088] At step S8, using the calculated camera transformations from step S6, further features are matched in the 
images (the calculated camera transformations behg used to calculate, that is "constrain", the position in an image in 
which to look for a point matching a given point in another image). This process will be described below with reference 
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to Figures 39 and 40. 

[0089] At step S1 0, points in a three-dimensional modelling space representing actual points on the surface of object 
24 are generated, as will be described below with reference to Figures 41 to 48. 

[0090] In step S12, the points in three-dimensional space produced in step S10 are connected to generate three- 
s dimensional surfaces, representing a three-dimensional model of object 24. This process will be described with refer- 
ence to Figure 49. 

[0091] In step S14, the 3D model produced in step S12 is processed to display an image of the object 24 from a 

desired viewing direction on display unit 18. This process will be described with reference to Figure 50. 

[0092] Figure 4 shows the steps performed in the initial data input routine at step S2 in Figure 3. Referring to Figure 

io 4, at step SI 6, the CPU 4 waits until image data has been received within memory 6. As noted previously, this image 
data may be received from digital camera 12, via floppy disk 1 0, by digitisation of a photograph using a scanner (not 
shown), or by downloading image data from a database, for example via a datalink (not shown), etc. 
[0093] After the data for all images has been received, CPU 4 re-stores the data for each image as a separate 
"project" file in memory 6 at step S18. At step S20, CPU 4 reads the stored data from memory 6 and displays the 

is images to the user on display unit 1 8. 

[0094] Figure 5 illustrates the display of the images to the user. CPU 4 initially displays the images in the order in 
which the image data was received. Referring again to Figure 2, images were taken from locations L1 , L2, L3 : L4 then 
L5. Accordingly, the image data of the images taken at these locations is stored in the same sequence within camera 
12 and is received by computer 2 in the same order when it is downloaded from camera 12. Therefore, as shown in 

20 Figure 5, CPU 4 initially displays the images on display 18 in the same order, namely L1, L2 : L3 : 14, L5. 

[0095] At the same time as displaying the images, CPU 4 prompts the user, for example by displaying a message 
(not shown) on display 18, to rearrange the images into an order which represents the positional sequence in which 
the images were taken around object 24, rather than the temporal sequence in which the images are initially displayed. 
The temporal sequence and the positional sequence may be the same. However, in the example illustrated in Figure 

25 2, location L3 is between locations L1 and L2. The positional sequence of images around the object 24 is, therefore,. 
L1, L3, L2, L4andL5. 

[0096] Accordingly, at step S22, the user rearranges the images on display 16, for example by highlighting the image 
taken at location L2 and dragging it to a position between the images tor positions L3 and L4 (as indicated by the arrow 
in Figure 5), to give the correct positional sequence for the images. 

30 [0097] Following this, at step S24, CPU 4 calculates the distance between the centres of the images on the display 
18 to determine the nearest neighbour(s) for each image. Thus, for example, referring to Figure 5. for the image taken 
at position Li. CPU 4 calculates the distance between its centre and the centre of each other image, and determines 
that the nearest image is the one taken at position L3 For the image taken at position L3, the CPU 4 calculates the 
distance between its centre and each of the images taken al positions L2, L4 and L5 (the CPU already having deter- 

3S mined that the image taken at position L1 is a nearest neighbour on one side of the image taken at position L3). In this 
way, CPU 4 determines that the image taken at position L2 is the nearest neighbour of the image taken at position L3 
on its other side. The CPU performs the same routine for the images taken at positions L2, L4 and L5. 
[0098] At step S26, CPU 4 stores links in memory 6 to identify the positional sequence of the images. For example, 
CPU 4 creates, and stores in memory 6, the links as separate entities. The data for each link identifies the image at 

40 each end of the link. Thus, referring to the example shown in Figures 2 and 5. CPU 4 creates four links, one having 
the images taken at positions L1 and L3 at its ends, one having the images taken at positions L3 and L2 at its ends, 
one having images taken al positions L2 and L4 at its ends, and one having images taken at positions L4 and L5 at 
its ends. 

[0099] At step S26, CPU 4 also stores in the project file for each image (created at step S18) a pointer to each link 
& entity connected to the image. For example, the project file for the image taken at position L3 will have pointers to the 
first and second links. 

[0100] At step S28, CPU 4 requests the user to input information about the camera with which the image data was 
recorded. CPU 4 does this by displaying a message requesting the user to input the focal length of the camera lens 
and the size of the imaging charge coupled device (CCD) or film within the camera. CPU 4 also displays on display 
50 1 6 a list of standard cameras, for which this information is pre-stored in memory 6, and from which the user can select 
the camera used instead of inputting the information directly. At step S30, the user inputs the requested camera data, 
or selects one of the listed cameras, and at step S32, CPU 4 stores the input camera data in memory 6 for future use. 
[0101] The processing of the image data stored in memory 6 by CPU 4 will now be described with reference to 
Figures 6 to 50. 

55 [0102] Figure 6 shows, at a top level, the relationship between the routines of initial feature matching, calculating 
camera transformations and constrained feature matching performed by CPU 4 at steps S4, S6, S8 in Figure 3. For 
the purpose of these routres, CPU 4 considers images in groups of three in the order in which they occur in the 
positional sequence created at step S22 (Figure 4), each group being referred to as a triple' of images. Thus, in the 
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case where data for five images has been stored in memory 6 (as in the example ol Figures 2 and 5), CPU 4 considers 
three triples of images (images 1-2-3, images 2-3-4, and images 3-4-5 in the positional sequence). Within each triple 
of images, there are two 'pairs' of images, namely the first and second images within the triple and the second and 
third images within the triple. 

s [0103] Referring to Figure 6, at step S40, the next triple of (mages is considered for processing (this being the first 
triple, that is images 1-2-3 in the positional sequence, the first time step S40 is performed). At step S42, initial feature 
matching is performed for the three images under consideration to match points across pairs of images in the triple or 
across all three images, and at step S44 the camera transformations between the positions at which the three images 
were taken are calculated using the points matched in step S42. The calculated camera transformations define the 

io translation and rotation of the camera between images in the positional sequence, as will be described in greater detail 
below. 

[0104] At step S46, CPU 4 determines whether the camera transformations calculated at step S44 are sufficiently 
accurate. If it is determined that the transformations are sufficiently accurate, then, at step $48, further features are 
matched in the three images using the calculated camera transformations. The feature matching performed by CPU 

'5 4 al step S4fi is termed 'constrained' feature matching since the camera transformations calculated at step S44 are 
used to 'constrain* the area within an image of the triple which is searched to identify a point which may match a given 
point in another image of the triple. II it is determined al step S46 that the calculated camera transformations are not 
sufficiently accurate, then steps S42 to S46 are repeated until sufficiently accurate camera transformations are ob- 
tained. However, as will be described below, when CPU 4 re-performs initial feature matching for the three images at 
. 20 step S42 for the first time after it has been determined at step S46 that the calculated camera transformations are not 
sufficiently accurate, it performs it using a second technique, which is different to the first technique used when step 
S42 is performed for the very first time. Further, in any subsequent re-performance of step $42. CPU 4 performs Initial 
feature matching using the second technique, but with a different number of matched points in the images as input 
(the number increasing each time step $42 is repeated). 

2$ [0105] At step S50, CPU 4 determines whether there is another image which has not yet been considered in the 
positional sequence of images, and, if there is, steps S40 to S50 are repeated to consider the next triple of images.' 
These steps are repeated until all images have been processed in the way described above. 
[0106] Figure 7 shows in greater detail the relationship between the routines of initial feature matching, calculating 
camera transformations and constrained feature matching. 

30 [0107] Referring to Figure 7, at slep S52, CPU 4 performs initial feature matching using a first technique for the first 
pair of images in a triple of images, as will be described below. This first initial feature matching technique is automatic, 
in the sense that no input from the user is required At step S54, CPU 4 performs initial feature matching using the 
first, automatic technique for the second pair of images in the triple. Al step S56, CPU 4 calculates the camera trans- 
formations between the images in the triple. At step S58, CPU 4 determines whether the camera transformations 

35 calculated at step S56 are sufficiently accurate. If they are, constrained feature matching is performed at step S74 to 
match further points in the images of the triple. 

[01 08] On the other hand, if is determined at step S58 that the calculated camera transformations are not sufficiently 
accurate, then CPU 4 performs initial feature matching for the triple of images using a different technique at steps $60 
to S68. In this embodiment, an "affine' technique (which assumes thai the object 24 in the images does not exhibit 

40 significant perspective' properties over smaQ regions of the image) is used, as wiD be described below 

[01 09] At step $60. the user is asked to identify matching points (that is, points which correspond to the same physical 
point on object 24) in the first pair of images of the triple and the second pair of images in the triple. This is done by 
displaying to the user on display unit 18 the three images in the triple. The user can then move a displayed cursor 
using input means 14 to identify a point in the first image and a corresponding, matched point (representing the same 
physical point on object 24) in the second image. This process is repeated until ten pairs of points have been matched 
in the first and second images. The user then repeats the process to identify ten pairs of matched points in the second 
and third images. It may be difficult for the user to precisely locate the displayed cursor at a desired point (which may 
occupy only one pixel) when selecting points. Accordingly, il any point identified by the user is within two pixels ol a 
point previously identified in that image by the CPU in step S52 or $54 or, if performed previously, in step S62, $64 or 

so S74, then CPU 4 determines that the user intended to identify a point which it had automatically identified previously, 
and consequently stores the co-ordinates of this point rather than the point actually identified by the user on display 16. 
[0110] At step S62, CPU 4 matches points in the first pair of images in the trple using the affine matching technique, 
and at step S64, it matches points in the second pair of images in the triple using this technique. As will be described 
below, in affine feature matching, CPU 4 uses the points matched by the user at step S60 to determine the relationship 

55 between the images in each pair of images, that is the mathematical transformation necessary to transform points from 
one image to the other, and uses this to identify further matching points in the images. 

[0111] At step S66, CPU 4 uses all of the points which have now been matched to determine again the camera 
transformations between the positions at which the three images in the triple were taken, and at step S68 determines 
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as a strong corner in the first image, in a conventional manner. At step S84. CPU 4 performs the operation previously 
carried out at step S80 tor the first image for the second image, and likewise identifies strong comers in the second 
image at step 536 us rig the same technique previously performed at step 582. 

[0119] At step S£6, CPU 4 compares each strong corner identified in the first image at step So 2 with every strong 

s corner identified In the second Image at step SS6 which lies within a given area centred on the pixel in the second 
image which has the same pixel coordinates as the corner point under consideration in the first image to produce a 
similarity measure lor the comers in the first and second images. In this embodiment, the size of the area considered 
in the second image is ±10 pixels of the centre pixel in the y-directton and ±200 pixels of the centre pixel in the x- 
direction. The use of such a 'window* area to restrict the search area in the second image ensures that similar points 

w which lie on different parts of object 24 are not identified as matches. The window in this embodiment is set to have a . 
small y value (height) and a relatively large "x" value (width) since it has been found that the images of object 24 are 
often recorded by a user with camera 12 at approximately the same vertical height (so that a point on Ihe surface of 
object 24 is not displaced significantly in the vertical (y) direction in the images) but displaced around object 24 in a 
horizontal direction In this embodiment, the comparison of points is carried out using an adaptive least squares cor- 

'5 relation technique, lor example as described in "Adaptive Least Squares Correlation: A Powerful Image Matching 
Technique" by A.W. Gruen in Photogrammetry Remote Sensing and Cartography 19S5 pages 175-187. 
[0120] At step 590, CPU 4 identifies and stores matching points. This is performed using a "relaxation* technique, 
as will now be described. Step S88 produces a similarity measure between each strong comer in the first image and 
a plurality of strong comers in the second image (that is. those lying wrthh the window in the second image described 

20 above). At step S90, CPU 4 effectively arranges these values in a table array, for example listing all of the strong 
corners in the first image in a column, all of the strong corners in the second image in a row, and the similarity measure 
for each given pair of comers at the appropriate intersection in the table. In this way, rows of the table array define the 
similarity measure between a given comer point in the first image and each corner point in the second image (the 
similarity measure may be zero if the corner in the first image was not compared with the comer in the second image 

25 at step S88). Similarly, the columns in the array define the similarity measure between a given corner point in the 
second image and each corner point in the first image (again, some values may be 2ero il the points were not compared 
at step SS8). CPU 4 then considers the first row of values, selects the highest similarity measure value in the row, and 
determines whether this value is also the highest value in the column in which the value lies. It the value is the highest 
in the row and column, this indicates that the comer point in the second image is the best matching point for the point 

30 in the first image and vice versa. 

[0121] In this case, CPU 4 sets all of the values in the row and the column to zero (so that these values are not 
considered in further processing), and determines whether the highest similarity measure is above a predetermined 
threshold (in this embodiment, 0.1). If the similarity measure is above the threshold, CPU 4 stores the point in the first 
image and the corresponding point in the second image as matched points. If the similarity measure is not above the 

35 predetermined threshold, then it is determined that, even though the points are the best matching points for each other, 
the degree of similarity is not sufficient to store the points as matching points. 

[0122] CPU 4 then repeats this processing for each row ol the table array, until all of the rows have been considered. 
If it is determined that the highest similarity measure in a row is not also the highest for the column in which it lies, CPU 
4 moves on to consider the next row. Thus, it is possible that no pairs of matching points are identified in step S90. 
-*0 [0123] CPU 4 reconsiders each row in the table array to repeat the processing above it matching points were identified 
the previous time all the rows were considered. CPU 4 continues to perform such iterations until no matching points 
are identified in an iteration. 

[0124] Figure 9 shows the steps performed by CPU 4 at step S54 in Figure 7 when performing automatic initial 
feature matching for the second pair of images in a triple. In this case, points in the first image of the pair have already 

4£ been identified: strong comers in steps S84 and S86 of Figure B when the previous pair of images was considered; 
and other feature points from automatic initial feature matching (step S54), affine initial feature matching (steps S60, 
S64 and S72) and constrained feature matching (step S74) if these steps have been performed for the previous triple 
of images. Accordingly, CPU 4 needs only to identify strong corners in the second image of the pair (the third image 
of the triple under consideration). 

50 [0125] Referring to Figure 9 al step S92, CPU 4 applies a pixel mask to the third image of the triple and calculates 
a value for each pixel in the third image indicating the amount of edge and comer for that pixel. This is performed in 
the same way as the operation in step S80 described previously. In step S94, CPU 4 identifies and stores strong comers 
in the third image. This is performed in the same way as step S82 described previously. At step S96, CPU 4 considers 
the strong points previously identified and stored at step S8S, S54, S60, S64, S72 and S74 for the second image in 

ss the triple and the strong corners identified and stored at step S94 for the third image in the triple, and calculates a 
similarity measure between pairs of points. This is carried out In the same way as step S88 described previously (again 
using a "window* to restrict the points in the third image which are compared against each port in the second image). 
At step S98, matching points in the second and third images of the triple are identified and stored This is carried out 
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in the same way as step S90 described previously. 
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sidering the first column of pixels in the area A, for example th© column of pixels which are left-most in the image. 
Within this column : it first considers the top pixel, and compares the edge magnitude and edge direction values calcu- 
lated at step S115 for this pixel against thresholds. In this embodiment the magnitude threshold is set at a very low 
setting of 0.01 smooth grey levels per pixel. This is because edges often become "weakened* in an image, for example 
s by the lighting, which can produce shadows etc. across the edge. Accordingly, by using a small magnitude threshold, 
it is assured that all pixels having any reasonable value of edge magnitude are considered. The direction threshold is 
set so as to impose a relatively strict requirement for the direction value of the pixel to lie within a small angular deviation 
(in this embodiment 0.5 radians) of the direction of the straight line connecting points 30 and 32. This is because 
direction has been found to be a much more accurate way of determining whether the pixel actually represents an 
io edge than the pixel magnitude value. 

[01 37] If the top pixel in a column of pixels has values above the magnitude threshold and below the direction thresh- 
old, then a Vote' is registered for that column, indicating that part or an edge between the points 30, 32 exists in (hat 
column of pixels. If the values of the top pixel do not meet this criteria, then the same tests are applied to the remaining 
pixels in the column, moving down the column. Once a pixel is found satisfying the threshold criteria, a Vote" is reg- 
is istered for the column and the next column of pixels is considered. On the other hand, if no pixel within the column is 
found which satisfies the threshold criteria, then no "vote* is registered for the column. When ad of the columns of 
pixels have been processed in this manner, CPU 4 determines the percentage of columns which have registered a 
"vote", this representing the strength ol the edge, and stores this percentage. 

[0138] Referring again to Figure 12, after performing steps S106 and S108, CPU 4 has calculated and stored a 

20 strength for each edge in each image of the pair. 

(0139] At step S1 10, CPU 4 calculates the combined strength of corresponding edges in the first image of the pair 
and the second image of the pair. This is done, for example, by reading the stored percentage edge strength calculated 
at step SI 06 for an edge in the first image and the value calculated in step S106 for the corresponding edge in the 
second image and calculating the geometric mean of the percentages (that is, the square root of the product of the 

25 percentages). If the resulting, combined strength value is less lhan 90%, CPU 4 determines that the edges are not 
sufficiently strong to consider further, and discards them. If the combined strength value is 90% or greater, CPU 4 
stores the value and identifies the edges in both images as strong edges (or future use. 

[0140] By performing step Si 10, CPU 4 effectively considers the strength of an edge in both images of a pair to 
determine whether an edge actually exists between given points. In this way an edge may still be identified even if it 
30 has become distorted (for example, broken) somewhat in one of the images since the strength of the edge h the other 
image will compensate. 

[0141] At step S112. CPU 4 considers the strong edges in the first image of the pair, that is the edges which remain 
alter the weak ones have been removed at step S1 1 0, and processes the image data to remove any crossovers between 
the edges. 

35 [0142] Figure 15 shows the operations performed by CPU 4 in determining whether any crossovers occur between 
the edges and removing them. Referring to Figure 15, at step S120, CPU 4 produces a list of the edges in the first 
image of the pair arranged in combined strength order, with the edge having the highest combined strength at the top 
of the list. Since the strength of the edges is calculated and stored as floating point numbers, it is unlikely that two 
edges wiD have the same combined strength. At step S122, CPU 4 considers the next pair of edges in the list (this 

^0 being the first pair the first time the step is performed), and at step S124, CPU 4 compares the coordinates of the points 
at the ends of each edge to determine "£»ether both end points of the first edge lie on the same side of the second 
edge. If it is determined that they do, CPU 4 determines at step S1 26 that the edges have a relationship corresponding 
to the case shown in Figure 16a and that therefore they do not cross. On the other hand, if it is determined at step 
Si 24 that both end points of the first edge do not tie on the same side of the second edge, then the edges have a 

<* relationship corresponding to either that shown in Figure 16b or that shown in Figure 16c. To determine which, at step 
S128 : CPU 4 again considers the coordinates of the points to determine whether both end points ol the second edge 
lie on the same side of the first edge. If they do, CPU 4 determines at step S1 26 that the edges do not cross, the edges 
corresponding to the case shown in Figure 16b. If it is determined that both end points ol Ihe second edge do not lie 
on the same side ol the first edge at step S128, then CPU 4 determines that the edges cross, as shown in Figure 16c, 

so . and at step S130 deletes the second edge of the pair, this being the edge with the bwer combined strength. This is 
done by setting the combined strength of the edge to zero, thereby effectively deleting the edge from both the first and 
second images. At step S132, CPU 4 determines whether there is another edge in the list which has not yet been 
compared. Steps SI 22 to SI 32 are repeated until all edges have been considered in the manner just described. That 
is, steps 122 to 1 32 are repeated to compare the edge with the highest combined strength with each edge lower in the 

ss list (proceeding down the list), and then to compare the next highest edge remaining in the list with each remaining 
lower edge (proceeding down the list) and to continue to compare edges in this decreasing strength order until all 
comparisons have been made (i.e. the next highest edge is the last in the list). 

[01 43] By arranging the edges in combined strength order at step S1 20, so that the edges are compared in this order, 
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where (x,y,i) are the homogeneous co-ordinates of a triangle vertex in the first image, the co-ordinate numbers Indi- 
es eating with which vertex the co-ordinates are associated, and (x\y\1 ) are the homogeneous co-ordinates of the point 
in the second image which is matched with the triangle vertex h the first image (again, the co-ordinate numbers indi- 
cating with which vertex the point is matched). This equation is solved in a conventional manner to calculate values 
for A to F and hence define the transformation for each triangle. 

[01 51 J At step Si 62 : CPU 4 divides the first image into a series of grid squares of size 25 pixels by 25 pixels, and 
20 sets a flag for each square to indicate that the square is 'empty*. Figure 1 9 illustrates an image divided into grid squares. 
At step Si 64, CPU 4 determines whether there are any points in the first image of the pair under consideration which 
have been matched with a point in the preceding image in the sequence but which have not been matched with a point 
in the second image of the pair. When the first image of the pair under consideration is the very first image in the 
sequence (the image taken at position 11 in the example of Figure 2) then there are no such points since there is no 
2S preceding image in the sequence. When the second image in the sequence (the image taken at position L3 in the 
example of Figure 2) is the first image in the pair under consideration, it will be seen from Figure 7 that points may 
have been matched with the preceding image (the first image in the sequence) by automatic initial feature matching 
at step S52, by user matching at step S60 or step S72 or by affine initial feature matching at step S62. When the first 
image of the pair under consideration is the third or a subsequent image in the sequence (one of the images taken at 
30 positions L2, L4 or L5), points may have been matched with the preceding image by Automatic initial feature matching 
at step S54, by user matching at step $60 or step S72, by affine initial feature matching at step S62 or step S64, or 
additionally by constrained feature matching at step S74. as described previously and as described in greater detail 
later 

[0152] Referring again to Figure 18, if CPU 4 determines at step S 164 that such points exist at step S166 rt considers 

35 one of the points, referred to as a "previously matched" point, and at step S16B determines whether this point lies 
within a triangle created at step S 102 in Figure 11 in the first image of the pair. If the point does not lie within a triangle, 
the processing proceeds to step SI 78 where CPU 4 determines whether there is another previously matched point in 
the first image of the pair. Steps S166, S166 and S176 are repeated until a previously matched point tying within a 
triangle in the first image of the pair is identified, or until all such previously matched points have been considered. 

40 When it is determined at step $1 68 that the previously matched point being considered does lie within a triangle in the 
first image of the pair, at step S1 70, CPU 4 tries to find a corresponding point in the second image of the pair. This is 
done by applying the affine transformation for the triangle in which the point lies (previously calculated at step St 60) 
to the co-ordinates of the point to identify a point in the second image, and then applying an adaptive least squares 
correlation routine, such as the one described in the paper "Adaptive Least Squares Correlation: A Powerful Image 

4s Matching Technique* by A.W. Gruen, Photogrammetry Remote Sensing and Cartography, 1985, pages 175-187, to 
consider the identified point in the second image and points in a small area around it to determine whether any point 
has the same image characteristics as the previously matched point in the first image of the pair. This produces a 
similarity measure for a point in the second image. At step SI 72. CPU 4 determines whether a corresponding point in 
the second image of the pair has been found by comparing the similarity measure with a threshold (in this embodiment, 

so 0.4). If the similarity measure is greater than the threshold, it b determined that the point in the second image having 
this similarity measure corresponds to the previously matched point in the first image and at step S1 74, CPU 4 changes 
the flag for the grid square in which the point h the first image lies to indicate that the grid square is full". At step S1 76, 
CPU 4 stores data identifying the points as matched. 

[0153] At step S178, CPU 4 considers whether there is another previously matched point in the first image of the 
ss pair not yet considered, and if there is, steps Si 66 to Si 78 are repeated until all previously matched points in the first 
image of the pair have been processed in the manner just described. 

[01 54] When all of the previously matched points in the first image of the pair have been processed, or if it is deter- 
mined at step S164 that there are no previously matched points, then at step S180, CPU 4 considers the next empty 
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34. 36, 38 in Figure ,9. than processin p^^^^S^u?^"^™™^ 
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proceeds to step S184 in which CPU J «Ss7he ooTwl L hI7 !- T *° Fi9 " r6 ,9) Processin 9 
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I01S4, F ^lshows.a,a,op,eve,.«^^ 
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step S202 in Figure 20. Referring to Figure 21, at step S2Q6, CPU 4 sets up the parameters necessary for the calcu- 
lation. At step S208, CPU 4 calculates the camera transformations between the first pair ol images in the triple and 
stores the results, and at step S210, CPU 4 calculates the camera transformations between the second pair of images 
in the triple and stores the results. At step S212, the camera transformations for the first pair of images calculated at 
step S20S and for the second pair of images calculated at step S2t0are used to calculate the camera transformations 
for all three images in the triple, these transformations then being stored. 

[01 65] Figure 22 shows the operations performed by CPU 4 in setting up the parameters at step S206. Referring to 
Figure 22, at step S214, CPU 4 reads the camera data input by (he user at step S30 (Figure 4). At etep.S2l6 s CPU 4 
reads the points matched in the first pair of images ol the triple during initial feature matching at steps S52, S60, S62 
and S72 (Figure 7) and the points matched in the second pair of images in the triple during initial feature matching at 
steps S54, S60, S64 and S72 (Figure 7). 

[0166] At step S218. CPU 4 generates, for each pair of images, a list of the matched points which are user-identified 
(that is, identified by the user at step S60 or S72 in Figure 7) and a list of matched points comprising both points 
calculated by CPU 4 as matching (at steps S52. S54. S62 or S64 In Figure 7) and user-identified points. Some of the 
calculated matching points may be the same as user-identified matching points. If this is the case, CPU 4 deletes the 
CPU-calculated points from the list so that there are no duplicate pairs ol matching points. By deleting the CPU-cal- 
culated points, CPU 4 ensures thai a point appears in both of the lists which will be ueed for the calculations (one of 
these lists being user-identified points alone, and hence the point would not appear in this list if user-identified points 
were deleted to remove duplicates). The number of points in the list of user-identified matching points may be zero. 
This will be case if affine initial feature matching at steps S60 to S72 in Figure 7 has not been performed. 
[0167] Also at step S21 8, CPU 4 generates a list of "triple" points, that is, points (including both user-matched points 
and CPU -calculated points) which are matched across all three images h the triple of images being considered. 
[0168] At step $220. CPU 4 normalises the co-ordinates of the points in the lists created at step S2 18. Up to this 
point, the co-ordinates of the points are defined in terms of the number of pixels across and down the image from the 
top (eft-hand corner of the image. At step S220, CPU 4 uses the camera focal length and image plane (film or CCD) 
size read at step S214 to convert the co-ordinates of the points from pixels to a co-ordinate system in millimetres having 
an origin at the camera optical centre. The millimetre coordinates are related to the pixel coordinates as follows: 



jT= h X (x-CJ 



-vX(y-C y ) 



where (x*,y*) are the millimetre coordinates, (x.y) are the pixel coordinates, (C r C y ) is the centre of the image (in pixels), 
which is defined as half of the number of pixels in the horizontal and vertical directions, and 'h' and "V are the horizontal 
and vertical distances between adjacent pixels (in mm). 
[0169] CPU 4 stores both the millimetre coordinates and the pixel coordinates. 

[0170] At step S222, CPU 4 sets up a measurement matrix, M, as follows for each of the list of user-identified points 
and the list of user-identified and calculated points generated at step S218: 



so 



M 



x 1 x 1 -y lXl x x -x x y x y x y[ -y[ x x -y x 1 

*2*i -y 2 *2 4 -^yz y^' yl x 2 -y 3 1 



\*k*k y^k *k -x k yL y&k 



..(5) 



where (x.y) are the pixel co-ordinates of the point In the first Image of the pair, (x\y*) are the pixel co-ordinates of the 
corresponding (matched) point in the second image of the pair, and the numbers 1 to k indicate to which pair of points 
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there are not sufficient user-identified points alone on which to perform the perspective calculation. 
[0180] On the other hand, if it is determined at step 5244 that the number of iterations is greater than zero, at step " 
S246 CPU 4 increments the value of a counter by one (the first time step S246 is performed, CPU 4 setting the counter 
value to one). At step 3248, CPU 4 selects at random seven pairs of points from the list of matched user-identified 
points set up at step S213 (Figure 22). At step S250, CPU 4 uses the selected seven pairs of points and the meas- 
urement matrix set at step S222 to calculate the fundamental matrix, F, representing the geometrical relationship be- 
tween the images, f being a three by three matrix satisfying the following equation: 



where (x,y,1) are the homogeneous pixel co-ordinates ol any of the seven selected points in the first image of the pair, 
and (x\y\1) are the corresponding homogeneous pixel co-ordinates in the second image of the pair. 
[0181] The fundamental matrix is calculated in a conventional manner, for example using the technique disclosed in 
'Robust Detection of Degenerate Configurations Whilst Estimating the Fundamental Matrix* by P.H.S. Torr, A Zisser- 
man and S. May bank, Oxford University Technical Report 2090/96. 

[0182] It is possible to select more than seven pairs of matched points at step S243 and to use these to calculate 
the fundamental matrix at step S250. However, seven pairs of points are used in this embodiment, since this has' been 
shown empirically to produce satisfactory results, and also represents the minimum number of pairs needed to calculate 
the parameters of the fundamental matrix, reducing processing requirements. 

[0183] At step S252, CPU 4 converts the fundamental matrix, F, into a physical fundamental matrix, F phy6 , using the 
camera data read at step S21 4 (Figure 22). This is again performed in a conventional manner, for example as described 
in "Motion and Structure from Two Perspective Views: Algorithms, Error Analysis and Error Estimation" by J. Weng, T. 
S. Huang and N. Ahuja, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 11, No. 5. May 1989, 
pages 451-476, and as summarised bebw. 

[0184] Firsl the essential matrix, E, which satisfies the following equation is calculated: 



where (x*. y*. f) are the co-ordinates ol any ol the selected seven points in the first image in a millimetre co-ordinate 
system whose origin is at the centre of the image, the z co-ordinate having being normalised to correspond to the focal 
length, f, of the camera, and (x*\ y*\ f) are the corresponding co-ordinates of the matched point in the second image 
of the pair. The fundamental matrix, F, is converted into the essential matrix, E, using the following equations: 



(x* y' l)f y = 0 



(6) 




...-..( 7 ) 



A ° 



\lh 0 c x /f y 
0 l/v -c y /f 
< 0 0 l/f j 



(8) 



M=A T FA 



(9) 




(10) 



17 



EP0 901 105 A1 



closest matrix being E phy J. ' " " ,,ans,a " on vect <» ■«« length) and rotation matrix (this 

[0186] Finaity, ,he physical essentia, matrbc is converted into a physical fundamental m a ,r«, using the equation: 



phr»- n c ptv$* (11) 
where the symbol '-V denotes the matrix inverse 

Led'ocateuLTefhe^^ 

distance (in the concatenated Ige^ 

representing the physical Cental maSx Ihk dlf a l T ""T^ 9 "* p8if * point8 ,ram ,he Efface 
aconventioLmLer.^e^^ 

the Fundamental Matrix" by RKs£ r * itf S S^ e ^ M * ,Eltal « 
[0190] Figure 26 shows the way in whtah cTuTte^h, nh !^ m ^ Un,Ve,Si,y TeChnica, He P° rt 209 <"9S- 
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Oxford University Technical Report 2030/96 unoamenlal bvp H.S. Torr, A. Z. S serman andS. Maybank, 
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tested), and (iii) CPU 4 calculates the total error for all points stored at step S300, using the following equation: 



Total error- 




where ej is the distance for the "ith pair of matched points between the 4D point represented by their co-ordinates and 
the surface representing the physical fundamental matrix calculated at step S296, this value being squared so that it 
J0 is unsigned (thereby ensuring that the side cf the surface representing the physical fundamental matrix on which the 
point lies does not affect the result), p being the total number of points stored at step S300 and e^ being the distance 
threshold used in the comparison at step S298. 

[01 94] In step S255. the counter value and stored points at step S300 {Figure 26) and the total error described above 
include the seven pairs of points tested at step S253 
»5 [0195] The effect of step S2S5 is to determine whether the physical fundamental matrix calculated at step S252 is 
accurate for each pair of user-identified and calculated points, the value ol the counter at the end (step S300) indicating 
the total number of the points for which the calculated matrix is sufficiently accurate. 

[0196] At step S256, CPU 4 determines whether the physical fundamental matrix tested at step S255 is more accurate 
than any previously calculated using the perspective calculation technique for the user -identified points alone. This is 

20 done by comparing the counter value stored at step S300 in Figure 26 for the last-calculated physical fundamental 
matrix (this value representing the number of points tor which the physical fundamental matrix is an accurate camera 
solution) with the corresponding counter value stored for the most accurate physical fundamental matrix previously 
calculated. The matrix with the highest number of points (counter value) is taken to be the most accurate. If the number 
of points is the same for two matrices, the total error for each matrix (calculated as described above) is compared, and 

& the most accurate matrix is taken to be the one with the lowest error. If it is determined at step S256 that the physical 
fundamental matrix is more accurate than the currently stored one, al step S258 the previous one is discarded, and 
the new one is stored together with the number of points (counter value) stored at step S300 in Figure 26, the points 
themselves, and the total error calculated for the matrix. 

[0197] At step S260, CPU 4 determines whether the value of the counter incremented at step S246 is less than the 

30 value "np" set Rt step S224 in Figure 22 defining the number of iterations to be performed. If the value is not less than 
"np", the required number of iterations has boon porformcd, and the processing procoods to stop S264 in ordor to carry 
out the perspective calculation for the points in the list comprising both user-identified points and calculated points. 
Alternatively, if the required number of iterations has not yet been reached (value of the counter is still less than Yip" 
at step S260) ; at step S262, CPU 4 determines whether the accuracy of the physical fundamental matrix (represented 

35 by the counter value and the total error stored at step S258) has increased at all in the last np/2 iterations. If it has, it 
is worthwhile performing further iterations, and steps S246 to S262 are repeated. If there has not been any change in 
the accuracy of the physical fundamental matrix in the last np/2 iterations, processing is stopped even though the 
number of iterations has not yet reached the value "np" set at step S224 in Figure 22. In this way, processing time can 
be saved in cases where performing the full number ol iterations would not produce significantly more accurate results. 

^0 [0198] As described above with respect to Figure 23, the value of 'np' is set based on the number of pairs of points 
in the list of points Irom which the seven pairs are selected at random at step S248. Referring to step S233 in Figure 
23, the value (k-1)(k-2)(k-3)(k-4)(k-5)(k-6)/20160 represents 25% of the maximum number of iterations that it would 
be possible to perform without repetition (this maximum number being the total number of different combinations of 
seven pairs of points selected from the list). The value np/2 used at step S262 has been determined empirically to 

-*£ produce acceptable results in a reasonable time. 

[0199] Referring again to Rgure 25 at steps S264 to S282, CPU 4 carries out the perspective calculation for the pair 
of images using pairs of points selected at random from the Ii3t comprising both user-identified and calculated points. 
The steps are the same as those performed at steps S244 to S262, described above, with the exception that the value 
*np a defining the number of iterations to be performed has been set differently (step S224 h Figure 22), and the seven 

so pairs of points used to calculate the fundamental matrix selected at random are chosen from the list comprising both 
user-identitied and calculated points. The operations performed in this processing will not therefore, be described 
again. As before, Figure 26 shows the steps performed when testing the physical fundamental matrix against each 
pair of user-identified and calculated points (step S273 and step $275). 

[0200] At step S264, CPU 4 compares the most accurate physical fundamental matrix calculated using the user- 
ss identified points alone (stored at step S258) and the most accurate physical fundamental matrix calculated using both 
the user-identified points and calculated points (stored at 6tep S278), and selects the most accurate of the two (by 
comparing the counter values which represent the number of points for which the matrices are an accurate solution, 
and, if these are the same, the total error). The most accurate physical fundamental matrix is then converted to a 
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[0210] At step S324, CPU 4 determines whether the value of the counter incremented at step S3 1 2 is less than the 
number of iterations, °na a , set for affine calculations on user-identified points alone at step S224 (Figure 22), and hence 
whether the set number o! iterations has been performed. If the value of the counter is not less than the set number 
of iterations, then the required number of iterations have been performed, and processing proceeds to step S329. If 

s the value of the counter is less than the set number of iterations, CPU 4 performs a further test at step S326 to determine 
whether the accuracy of the affine fundamental matrix has increased at all in the last na/2 iterations. If the accuracy 
has not increased, then processing is stopped even though the set number of iterations, "na", has not yet been per- 
formed. In this way, iterations which would not produce any increase in the accuracy of the affine fundamental matrix 
are not performed, and hence processing time is saved. On the other hand, if the accuracy has increased, steps S31 2 

io to 5326 are repeated until either it is determined at step S324 that the set number of iterations has been performed 
or it is determined at step S326 that there has been no increase in accuracy of the affine fundamental matrix in the 
previous na/2 iterations. 

[0211] At step $327, CPU 4 converts the stored affine fundamental matrix (that is, the most accurate calculated using 
the user-identified points alone) into three physical variables describing the camera transformation, namely the mag- 

i5 nification, "m\ of the object between the two images, the axis, of rotation of the camera, and the cyclotorsion rotation, 
6. of the camera. (The variables $ and 9 will be described in greater detail later.) The conversion of the affine fundamental 
matrix into these physical variables is performed in a conventional manner, for example as described in 'Affine Analysis 
of Image Sequences" by LS. Shapiro. Section 7, Cambridge University Press, 1995, ISBN 0-521-55063-7. 
[021 2] In steps S323 to S345, CPU 4 carries out the affine calculation using pairs of points selected at random from 

zo the list containing both user-identified points and calculated points. The steps are the same as those performed by 
CPU 4 for user-idenlified points alone in steps S310 to S327 descrbed above, with the exception that the number of 
iterations, 'na*, may have been set to a different value at step S224 in Figure 22, and the four pairs of points selected 
at random at step S332 are selected from the list comprising both user-identified and calculated points. These steps 
will therefore not be described again. 

2S [0213] Having performed the affine calculation using pairs of points from the list containing user-identified points 
alone (steps S310 to S327) and using pairs of points from the list comprising both user-identified and calculated points 
(steps S328 to S345) producing an affine fundamental matrix and which is the most accurate for each calculation, at 
step S346, CPU 4 compares these two affine lundamental matrices and selects the most accurate, this being the one 
having the highest number of points (stored at steps S322 and S340), and if the number ol points is the same, the one 

30 having the lowest matrix total error 

[0214] Referring again to Figure 21, having calculated at step S208 the camera transformation for the first pair of 
images in the triple using the perspective and affine techniques described above, and having calculated at step S210 
the camera transformation for the second pair of images in the triple using the same perspective and affine techniques, 
at step S21 2 CPU 4 uses the results to calculate the camera transformations for all three images in the triple together. 

35 [021 S] Figure 28 shows the operations performed by CPU 4 in calculating the camera transformations for all three 
images in the triple together at step S212. 

[021 6] When considering all three images in the triple, there are two camera transformations - one from the position 
at which the first image in the triple was taken to the position at which the second image was taken, and one from the 
position at which the second image was taken 1o the position at which the third image in the triple was taken. Each of 
40 these transformations can be either an affine transformation or a perspective transformation, giving four possible com- 
binations between the images (namely affine-affine. affine-perspective : perspective-affine and perspective-perspec- 
tive). Accordingly, at steps S350, S352, S354 and S356, CPU 4 considers a respective one of the four possible com- 
binations, and at step S358 selects the most accurate solution from the four. This processing will now be described in 
greater detail 

4* [0217] At step S350, CPU 4 considers the case in which the transformation between the first pair of images in the 
triple is affine : and the transformation between the second pair of images is also affine. Previously, at step S208 (Figure 
21) CPU 4 has already calculated the affine fundamental matrix and associated three physical variables defining the 
affine transformation between the first pair of images in the triple. Similarly, at step S210 (Figure 21) CPU 4 has cal- 
culated the affine fundamental matrix and associated three physical defining the affine transformation between the 

so second pair of images in the triple. As noted previously, the three physical variables derived from an affine fundamental 
matrix do not fully define the movement of the camera between a pair of images. At step S350, CPU 4 uses the 
previously calculated three physical variables to calculate the parameters necessary to define fully the camera move- 
ment between each pair of images. 

[0218] Figures 29a and 29b illustrate the parameters which it is necessary to calculate at step S350 to define fully 
ss the camera movements. Figure 29a shows a CCD imaging device, or film, 50 on which the images are formed in three 
different locations and orientations, representing the locations and orientations at which the first, second and third 
images in a triple were taken. Lines 52 represent the optical axis of the camera 12. The optical axis 52 moves a distance 
dl in moving from the first position to the second position, and a distance d2 in moving from the second position to the 
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third position. 
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from the second image intersect the ray from the first image at a point 60. This calculation is performed by CPU 4 as 
follows: 

a) The sign of p1 is flipped (reversed) if sin(p1 ) x sin($1 )>0. This is done because of prior knowledge ol the ordering 
5 of the images. 

b) The rotation matrix, R. is defined from the angles (91. $1, p1 ) using the equations: 

i 0 fl^f+Msinp+M^I-cospHflg (13) 



0 0 
0 0 
t -sin<J> cos<j> 



sin<J> ^ 
-cos<|> 

0 , 



(14) 
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R Q - J+Xs/ne+X 2 (1-costJ) 



(15) 



2$ 



0 -1 0 > 
10 0 

1,0 0 0; 



[16) 



where I is the identity matrix. 

c) The translation vector, t, from the point position in the two images the rotation matrix, R, and the change 
in magnification between the two Images, "m*. are defined using the equations: 
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r =(/^1.m(/? M + ^ r fl to p) r 



(17) 
(13) 
(19) 
(20) 
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[0234] Similarly, at step S400, CPU 4 varies the translation of the camera between the second and third images to 
make the ray from the third Image intersect the ray from the second Image at a point 62. 
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a projected point 70 in the second image (Figure 35). 

[0243] At step S432, CPU 4 calculates the distance, Y, between the projected point 70 in the second image and 
the actual point 72 in the second image from the triple of points read at step S424. At step S434, CPU 4 determines 
whether the distance calculated at step S432 is less than a threshold: set at 3 pixels in this embodiment. The closer 

5 together the projected potnl 70 and the actual point 72 In the second image, the mora closely this triple of points 
supports this value for the scale calculated at step S402 (Figure 32). Accordingly, if the distance is below the threshold, 
the calculated scale is considered to be sufficiently accurate, and at step S436. CPU 4 increments the variable P 
representing the number of triple points for which the scale is accurate, notes the points in the triple under consideration 
as being accurate for the scale under consideration, and updates the total distance error (that is, the error for all the 

io points so far tor which the distance calculated at step S432 was deemed to be below the threshold at step S434) with 
the new distance calculated at step S432. The total error is calculated using the following equation: 



where e { is the distance between the projected point 70 and the actual point 72 in the second image for the "i*th triple 
of points, this value being squared so that it is unsigned (thereby ensuring that only the magnitude of the distance 
20 between the projected point and the actual point is considered, rather than its direction, too), P being the total number 
of points, and e^ being the distance threshold used for the comparison at step S434. 

[0244] On the other hand, if it is determined at step S434 that the distance is not below the threshold, step S435 is 
omitted so that the variable P is not incremented. 

[0245] At step S438, CPU 4 determines whether there is another triple of points in the list generated at step $218 
25 (Figure 22). Steps S424 to S438 are repeated until the processing described above has been carried out for all the 
triple points in the list. At this point, the value of the variable P then indicates the total number of triple points for which 
the calculated scale is sufficiently accurate. 

[0246] Referring again to Figure 32, after testing the scale at step S404 using the method just described, CPU 4 
determines at step S406 whether the calculated scale is more accurate than any currently stored. This is done by 

30 comparing the number of points, P, and the total error stored at step S436 (Figure 34) with the number of points and 
total error for the previously stored best scale so far. The most accurate scale is tho one with tho largest number of 
points or, if the number of points is the same, the one with the smallest total error If the newty calculated scale is more 
accurate, then it, the number of points, P, and the total error are stored at step S408 to replace the previous most 
accurate scale, number of points, and total error. If it is not then the previous most accurate scale, number of points, 

35 and total error are retained. 

[0247] At step S4 10, CPU 4 determines whether the value of the counter incremented at step S392 is less than 20. 
If It Is, at step S412, CPU 4 determines whether there is another triple of points In the list stored at step S218 (Figure 
22). Steps S392 to S412 are repeated until twenty triples of points have been used to calculate the scale (determined 
at step S410) or until all the triples of points in the list stored at step S218 (Figure 22) have been used to calculate the 

40 scale (determined at step S41 2) if the number of triple points is less than 20. The value 20 has been found empirically 
to produce acceptable results for the scaJe calculation in a reasonable time. 

[0248] Referring again to Figure 30, after calculating at step S382 the best value of the scale for the value of p1 , p2 
under consideration, at step S384, CPU 4 determines whether the solution, that is, the values of p1, p2, s are more 
accurate than the solution currently stored. Thus, CPU 4 tests whether the latest values pi , p2, s calculated at steps 

4£ 5330 and S382 have produced more accurate camera transformations than values which were previously calculated 
at steps S380 and S382. This is done by comparing the number of points, P, stored for the current most accurate 
solution and stored for the latest solution at step S408 (Figure 32) and step S436 (Figure 34). The most accurate 
solution is the one with the highest number of points, or the one with the smallest total error if the number of points is 
the same. If the new solution is more accurate than the currently stored solution, then at step S386, CPU 4 replaces 

so the currently stored solution with the new one. On the other hand, if the currently stored solution is more accurate, it 
is retained. 

[0249] At step S388. CPU 4 determines whether there is a further value of p1, p2 to consider, and steps S380 to 
S388 are repeated until all values of pi, p2 have been processed as described above. Referring to Figure 31 again, 
it will be seen from Figure 31a that steps S380 to S338 will be performed sixty four times for the affine-affine case 
ss calculation at step S350 (Figure 26). It would also be appreciated from Figure 31b and Figure 31 c that steps S380 to 
S3S6 will be performed eight times for the affine-perspective case calculation at step S352 (Figure 28) and eight times 
for the perspeclive-afiine case calculation al slep S354 (Figure 28). Steps S360 to S386 wilt be performed only once 
for the perspective-perspective case calculation at step S356 (Figure 28) since, as shown in Figure 31 d, only one value 



is 



Total error = 




(29) 
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of pi. p2 is available for consideration at step S380. 
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above with respect to step S218 in Figure 22}, and a list of triple points, that is, points which are matched across all 
three images in the triple of images. (Note that step S54 or S64 may match a point in the third image of the triple with* 
a point in second image of the triple which was previously matched with a point in the first image of the triple by 
constrained feature matching at step S74 in Figure 7. In this case, the points identified by constrained feature matching 
s will form part of a triple of points, which will bs used in calculating the camera positions at step S404, and possibly 
step S394, if selected). As noted above with respect to step S218 in Figure 22, the number of user-identified points 
may be zero if affine initial feature matching has not been performed. 

[0259] At step S466 : CPU 4 normal is sg the points in the lists created at step S464. and at step $468, sets up two 
measurement matrices; one for the list of user-identified points and one for the list of user-identified and calculated 

io points. These steps are carried out h the same way as steps S220 and S222 in Figure 22 described above, and 
accordingly will not be described again. At step S470, CPU 4 determines the number of iterations to be performed 
when carrying out the perspective and affine calculations for the second pair of images in the triple. This is performed 
in the same way as step S224 in Figure 22 described above, and accordingly will not be described again. 
[0260] Referring again to Figure 36, having set up the necessary parameters at step S450, at step S452, CPU 4 

is calculates the camera transformation for the second pair of images in the triple and stores the results. This is carried 
out in the same way as step S208 or S 210 in Figure 21 described above, and accordingly will not be descrbed again. 
[0261] At step S454, CPU 4 uses the camera solutions for the first pair of images read at step S460 (Figure 37) 
together with the camera transformation calculated at step S452 for the second pair of images in the triple to calculate 
camera transformations between all three images in the triple. 

20 [0262] Figure 38 shows the operations performed by CPU 4 when calculating the camera transformations between 
the three images in the triple at step S454 in Figure 36. These operations are very similar to those performed in step 
S21 2 (Figure 21 ), and described above with respect to Figure 28, when calculating the camera transformations between 
the first three images in the positional sequence. As noted above, the relationship between the cameras for the first 
pair of images in the triple is already known from calculations on the preceding triple. It is therefore necessary to 

25 consider the transformation between only the second pair of images. Accordingly, at step S472, CPU 4 considers the 
case where the transformation between the second pair of images is affine. This is done by considering the camera 
solution for the first pair of images (read at step S450 in Figure 36) together with the most accurate affine fundamental 
matrix calculated for the second pair of images in step S452 (Figure 36). and calculating the scale, s, and p2 using the 
same operations described above with respect to step S354 in Figure 28. 

30 [0263] At step S474, CPU 4 considers the case where the transformation between the second pair of images is 
perspective. CPU 4 use3 the calculation for the first pair of cameras read at step S460 (Figure 37) together with the 
most accurate rotation matrix and translation vector for the cameras for the second pair of images obtained in step 
S452 (Figure 36) to calculate the scale using the same operations as in step S356 (Figure 28). In steps S476 to S488, 
CPU 4 carries out processing which is the same as that carried out at steps S356 to S370 in Figure 28, described 

35 above. That is, CPU 4 selects the most accurate solution from the one calculated at step S472 and the one calculated 
at step S474, and determines whether this is sufficiently accurate or not, optimising it if necessary at step S486 (which 
corresponds to step 5368 in Figure 28) (it being noted that the solution is not optimised if it is determined at step S484 
that the solution corresponds to the '-perspective case since the values of p are optimised and, in the perspective 
transformation for the second pair of images, p is already sufficiently accurate since it is defined in the calculated 
fundamental matrix, and the value of p for the first pair of images will either be defined in a fundamental matrix H the 
transformation is perspective or will already have been optimised at step S368 in Figure 28 if the transformation is 
affine). 

[0264] Referring again to Figure 7. a description will now be given of the way in which CPU 4 performs constrained 
feature matching for a triple of images at step S74. 
4& [0265] Figure 39 shows, at a top level, the operations performed by CPU 4 when carrying out constrained feature 
matching. 

[0266] Referring to Figure 39, at step S500, CPU 4 considers 'double 1 points in the first pair of images in the triple, 
that is points which have been matched between the first pair of images at step S52, S54, S60, S62, S64, S72 or S74 
(steps S54, S64 and S74 being applicable if performed for a previous triple of images) in Figure 7, but which have not 
$0 been matched between the second and third images h the triple. For each pair of such "double* points, CPU 4 tries 
to identify the corresponding point in the third image. If it is successful, a triple of points, (that is, points matched across 
all three images) is created. 

[0267] Similarly, at step S502, CPU 4 considers 'double' points h the second and third images of a current triple 
(that is, points which have been matched across the second pair of images at slep S54, S60, S64 or S72 in Figure 7, 
ss but which have not been matched across the first pair of images in the triple) and tries to identify a corresponding point 
in the first image to create new triples of points. 

[0268] Figure 40 shows the operations performed by CPU 4 at step S500 and at step S502 in Figure 39. Referring 
to Figure 40, at step $504, CPU 4 considers the next point in the second (centre) image of the triple which forms a 



27 



EP0 901 105 A1 

•double 



>"P'°CPU4ca,cu,a.es,hes;^ 

"> as that described in the paper Votive L^J^^^^l^JT^ ,ore «rnple SU ch 
Gruen. Phc.cgrammetry^emote S^sin 2££* S Da l 8 ^^7^ "f"" 9 TeChni <^ «* * * 
[0270] At step S510. CPU 4 determines wheSSJ^' P 9 8 '° ' dan,lfy a ' best mateh " 

S506 is greater than a threshold (J ZS£S£, To nUH^ZTZ* " P ° in ' identified a < *«P 

4 determines that the similarity LitSK^ "IT* * 9 ' ea,9r than ,he lh ' Mh «** C PU 

« triple fa sufficiently high to consider th polT T™tc ng poi Tand^ ? ^"'J" ,h9 remaWn9 imaoe 01 « ha 
the -double- points and the new point identified J °tZe™£^ n Tnl ? ^ ,0ma a ,riple °' P° int8 ««n 
CPU 4 determines at step S510 mat the «ZtotomL«^ 9 , 9 * ' he ^ °' lma » M - 0n < he «*w hand, if 

processed in the manner described above P ™ ** ,he °' ,ma9es bein 9 ^nsidered have been 

S500 in Figure 39) and new fimtth^bZZ^^Z Z« T h " QM °' 3 ,riple * *W (itop 

* These new matches are used by CPU 4 to generate L^J!1 "TT * *• ,riP ' 9 (8t9p S502 h Fi 9 ure 39 > 
described below In addition, however ^efelg to fLu^ ** * St9P 810 in Fi ° u '° 3 - « 

pair of images in a triple are taken into accoZ 2l,n ^ ! T 9Sn8ra,ed be,W8en P° in,s in »» *°cond 
^because, as explained pr^ 

matches for the second pair of images in am^TJ^ZT 9 'T?** °* " St6p 874 10 " en % new 

* a^sconsio-ared.andbtfh^ 

matching performed a. step S64 attempt to match oonts ^ ,ln " erfom f als ' B PS54 B ndth efl ffineini, ta |,e a ,u ra 
previous* been matched across X^^SSS.^ !T °^ 9M Wple h ™ 
images ^ed during constrained fea.^ **«• the first pairof 

performing initial feature matching for the next trtote « 9 8 29) ?° taken in, ° ""^ion whan 

* 4 9™ates,hemree^imensionalda^ 

matching is carried out at step S74 in Ficure 7 h* h» i£5T betow. When constrained feature 

ofimagestobecc^idered.andac orS^ 

are not taken into consideration during toKl^^££^*^ ' h9 ^ pair ot ima 9 es in triple 

- KTRe^rinTa^^ 

formations (step sVand pe^ •* the camera trans- 

uses the resuHs to general 3D data aTsTeTsfo To aim ^ ' ^ 38 " ^ mam8r deSCribed **»•• CPU 4 
three-dmensional space correctly pwton- to rUHT . T"" " * " n9fa,a 8 8ingl8 "* of P ointe " ■ 
[0274] Figure 41 shows the 0,1^^1 ^TZ^ 9 °° ** 8UrfaCB of 1,18 °*«« 24 - 
« KefJing .^Figure 4,°a7s« t ep e S^Z^^^*??* ■» » - a, step S,0 in Figure 3. 
Figures 2 and 5, the pairs comprising U L3 iJESSt^i JET S9qU6nCe h *™ 6Xample ot 

a user-identified "double" of pLs "that is ^^^ ^^2°™!°™ "'^ *° pair <°™ 
S60 or S72 in Figure 7 but no. matched wtth t°' iW98S ^ US8ralsls P 
pair of images) or pari of a triple of poinle with a «J^?E2LT7 ^ y - P 9CBd,nB ° r immedia ""V «°"owing the 
» ^^CPU^betweentheiriesinrpr^dbSween^ 

the positional sequence, to calubte a s^g.e^oinTLTlceT^ ° Fa !, and SUbS9quent in 

considers only pairs of matched points which (LZ J^^T. * h P air rt P*'^- h step S520, CPU 4 
transformaticfl when this transflatioT^ 

pants when constrained feature matching was ™Z™h fl t 1 9 3 ' Were ,den,ifi6d 34 new ™"<:hing 
• < r a pair to a tripte during oonSSZK S «Z« hSSTS * °' ^ ^ 
feature matching which were not considered to be JSS£ , .l' ^ initial 

not considered by CPU 4 in step S520 (unless Ihe^n^^!.^ ^ calculated camera transformation are 
matching). ( unless mey were subsequently extended to a triple by constrained feature 



28 



EP0 901 105 A1 



[0275] Figure 42 shows the operations performed by CPU 4 when calculating the 30 points at step S520. Referring 
to Figure 42, at step S530, CFU 4 considers the next pair of images in the sequence (the first pair whan step S530 is 
performed for the first time). At step S532, CFU 4 projects from each point in the next pair of points in the pair of images 
considered at step S530 which is either a point from a user-identified 'double' or a point from a triple of points, a line 
s in three-dimensional space through the optical centre of the camera for that point. This produces rays similar to those 
shown in Figure 35, with the exception that the rays are projected from adjacent images in Figure 35 since the images 
are considered in pairs. 

[0276] At step S534, CPU 4 calculates the mid-point of the line segment which connects, and is perpendicular to, 
both the lines projected in step S532 (this mid-point corresponding to the point 63 shown in Figure 35, and representing 

10 a physical point on the surface of object 24). At step SS36, CPU 4 determines whether a corresponding point has been 
matched in the next image of the sequence, that is, whether the points from which rays were projected in step S532 
form part of the triple of points with the subsequent image. If it is determined that a corresponding point has been 
matched in the next image, CPU 4 projects a line from the matched point h the next image in the same way that it did 
from the points in step S532. At step S540, CPU 4 calculates the mid-point of the line segment which connects, and 

'£ is perpendicular to the new line projected at step S538 and the line projected from the point in the previous image at 
step S532. in the same way that the mid-point is calculated in step S540. 

[0277] At step S542, CPU 4 determines whether a corresponding point has been matched in the next image of the 
sequence. Steps S533 to S542 are repeated until the next image in the sequence doss not contain a corresponding 
matched point or until all the images in the sequence have been processed. 

20 [0278] By way of example, referring to a sequence of images containing five images, such as the example shown 
in Figure 2 and Figure 5. steps 5532 and S534 will project a ray from a point in the first image and a matched point in 
the second image and calculate a single three-dimensional point (the mid-point in step S534) which represents the 
projection of the point in the first image and the point in the second image. Thus, a single point in three-dimensional 
space representing a physical point on the surface of object 24 is obtained from a pair of points between adjacent 

zs images in the sequence. If the third image hi the sequence contains a point which is matched to those in the first and 
second images (determined at step $536), steps S538 and S540 project a tine from the point in the third image and 
calculate the mid-point of the line segment which connects, and is perpendicular to, the line from the point in the second 
image and the line from the point in the third image, this mid-point representing the 3D point resulting from the projection 
of the points in the second image and third image. Similarly, it the fourth image in the sequence has a point matched 

30 to that in the Ihird image (determined at step S542), steps S53B and S540 are repeated to project a line from the point 
in the fourth image and calculate the mid-point of a line segment which connects, and is perpendicular to, the line from 
the fourth image and the line from the third image. A further 3D point representing the projection of points from the 
fourth and fifth images in the sequence will be obtained by step S538 and S540 if it is determined at step S542 that a 
corresponding point has been matched in the fifth image of the sequence. Thus, if the point is matched in all five images 

35 of the sequence, four 3D points are produced (representing the same physical point on the surface ol object 24), 
although it is unlikely that the 3D position of these will be exactly coincident due to errors in the calculated camera 
transformations and the matches themselves. Instead, the points form a cluster 60 in 3D space, as shown in Figure 43. 
[0279] Referring again to Figure 42, at step S544, CPU 4 determines whether there is another pair of points not 
previously considered in the current pair of images which form a user-identified 'double" of points across the pair of 

40 images or form part of a triple of points with a subsequent image. Steps S532 to S544 are repealed until all such points 
have been considered. Each such pair of points produces either a single point 82 in 3D space (Figure 43) if it is 
determined at step S536 that a corresponding point has not been matched in the next image or a cluster of points if 
the corresponding point has been matched in at least the next image. If the point is matched across three successive 
images in the sequence, the cluster contains two points, if it is matched across four successive images in the sequence 

4£ it contains three points, and., as described above, if it is matched across five images in the sequence, the cluster 
comprises four points as shown in cluster 80 ol Figure 43. 

[0280] At step S546. CPU 4 considers whether there is another pair ol images in the sequence. Steps S532 to S546 
are repeated until all pairs of images in the sequence have been processed as described above. The result is a plurality 
of clusters of points in three-dimensional space as shown in Figure 43, with the points within each cluster corresponding 
so to what should be a single 3D point (this representing a point on the surface ot object 24). 

[0281] Referring again to Figure 41, at step S522, CPU 4 uses the 3D points calculated at step S520 to calculate 
the error in the transformation previously calculated for each camera, and to identify and discard inaccurate ones of 
the 3D points. 

[0282] Figure 44 shows the operations performed by CPU 4 at step S522 in Figure 41. Referring to Figure 44, at 
55 step S550, CPU 4 considers alt of the points in three-dimensional space calculated at step S520 (Figure 41) and 
calculates the standard deviation of the x co-ordinates, Ax, the standard deviation of the y co-ordinates, Ay, and the 
standard deviation of the z co-ordinates, Az. At step S552, CPU 4 calculates the "size* of the object made up of the 
points in the three-dimensional space using the formula: 
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} (31) 

raining after^accuratepdn^ 
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at step S558 (Figure 44) that the magnitude of th 2 h «r!f ^ '° Fl9Ure 4Sa ' if jt ia ^termined 
96wc.,db e diecarded.^meo*erLnd Ifti ^ 

than 10% of the object size, the points are 'cwttSSlt^^ ma9n ' tUda ^ 0,9 6hift ie not 9 r9ate ^ 
as noted above, 3D points are no^a* a S 

to be accurate with the calculated LnWSSXS ^^^"""P^^^nte which were not considered 
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in Figure 44 ensure* that the accuracy «to£ZX£EZ^ " * " y CPU 4 in S,8ps 5554 to S56 ° 

s,onXacrjr:^^^ 

points which formed par, cf a triple with a £Z. S ££ t £T *"* 0rifllr,ated "° m 

have been processed as described above Fiqure45b 2. ,Z f ^ 1 2 ™ ' 9p9ated un,il a " euen P° int9 
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firs, pair of irnages and toe corresponding p ^ 

calculated. If any shift is greater than W. atheX £ 2^1! !k 9qUBm "* 01 ima " S wi " ^ b99n 
andthe point for the subsequent (second) ^SZ^^JT^ tm ^ ) ^ mw ^^ 
^'™no shift is ca.culatl^^ 

two successive images, and hence i, is no, posS to ^ ™«**««- only 

space which was derived from the corresponding poirtmS ?„ *1 8X,St8 h 1,16 lh ' e9 ^™n S ional 

[0287] Referring again to Figure 44 i£^£to?£!^f ? ««»mi» wage of ,he sequence. 

the current pair of JnJ^SZ^Ji'SSiS^!^^ °' a " B " 8hi " 8 b8 '" 8en the poinl » te ' 

gr«terman,0 % of,hec*£ e 4re ( d"er m h^ 

and an error translation vector for the subsequent pair afalU J££ ^ ^ *" 8TOf ro,a,ion "»'* 
conventional manner, for example using Hom'sm^^fl^ ^ u ^ " 9t °' the ahm b calcula '^ * a 
Orientation using Unft QuatZton^by STp Hot ? ^^f^^f^^^^^^^SoMkx, of Absolute 
1987. ,n summary,' the robtion ma«£ B R i ^SlJr^Tff^ l^ 9 * 3 ^ ^ ^ 
subsequent pair of camera positions to the corresD^r^l-* ' aCCUrat8ly maps lh9 point8 ,or *° 
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and P n * is the re-mapped version of F,,, then: 



P n=*P„+l (32) 

s 

[0288] The sum is minimised over all common points of the modules of the dot product (Pn-p€) T# (Pn'* p e)- 
[0289] At step S566, CPU 4 applies the error rotation matrix and the error translation vector calculated at step S564 
to each point previously calculated for the subsequent pair of camera positions (#2 in Figure 45b). For each previously 
calculated point, this gives a corrected point (P n ' given by Equation 32 above) which is now positioned closer to the 
10 point for the currenl pair of camera positions, as shown in Figure 46. in which the points for the current pair of camera 
positions are represented by dots as before, and the corrected points for the subsequent pair of camera positions are 
represented by crosses. 

[0290] At step S55B, CPU 4 calculates the' difference between the co-ordinates of each corrected 3D point calculated 
at step S566 and its corresponding point, and calculates the cc-variance matrix of the resulting differences, this being 
15 performed using conventional mathematical techniques. The resulting co-variance matrix comprises a Gaussian dis- 
tribution in three dimensions, which represents a three-dimensional error ellipsoid lor the error transform calculated at 
step S564. Thus, in steps SS64 to S568, CPU 4 has calculated an error transform for the subsequent pair of camera 
positions and the error (the error ellipsoid) associated with the error transform. 

[0291] At step S570, CPU 4 determines whothor there is another pair of camera positions which has not yot boon 
20 considered. Steps S554 to S570 are repeated until the data lor all pairs of camera positions has been processed in 
the manner described above. 

[0292] It will be appreciated that an error transform is not calculated at step S564 for the first pair of camera positions 
in the sequence. This pair of camera positions is assumed to have zero error. It will also be appreciated that the error ■ 
transform for a given pair of camera positions is calculated relative to the previous pair of camera positions. Thus, the 
error transform for the second pair of camera positions (that Is, producing the second and third Images in a sequence) 
includes no cumulative error since the error for the first pair of camera positions is assumed to be zero. On the other 
hand, the error transform lor each subsequent pair of camera positions will include cumulative error. For example, the 
error transform for the third pair of camera positions (that is, tho positions producing the third and fourth images in tho 
sequence) is calculated relative to the error transform for the second pair of camera positions. Accordingly, the calcu- 
lated error transform and co-variance matrix for the third pair ol camera positions needs to be adjusted by the error 
transform and co-variance matrix for the second pair of camera positions to give a total, cumulative error for the third 
pair of camera positions. Similarly, the calculated error transform and co-variance matrix for the fourth pair of camera 
positions (producing the fourth and fifth images in the sequence) needs to be adjusted by the error transform and co- 
variance matrix for both the second pair of camera positions and the third pair of camera positions (that Is, the cumulative 
error for the third pair of camera positions) to give a total, cumulative error for the fourth pair of camera positions. 
[0293] This is carried out by CPU 4 at step S572 as follows: 

R { = R hi R, (33) 
'/=*A-i+'/ (34) 



25 



C'i' E c « (35) 



o-l »i 

where R/ is the rotation matrbefor the ith cumulative error transform, is the rotation matrix for the ith individual error 
transform, if is the translation vector for the ith cumulative error transform, tj is the translation vector lor the ith individual 
error transform, C{ Is the covariance matrix for the ith cumulative error transform, and C n is the covariance matrix for 
the nth individual error transform. 

[0294] Referring again to Figure 41 , after calculating the error for each pair of camera positions at step S522, at step 
S524, CPU 4 adjusts the co-ordinates of each remaining point in the three-dimensional space (that Is, the points cal- 
culated at step S520 less those discarded at step S560 in Figure 44) by the appropriate camera position error. This is 
done by applying the cumulative error transform (calculated previously at step S572 in Figure 44) to the point position 
and adding the appropriate error ellipsoid (also previously calculated at step S572 in Figure 44) to the point. For ex- 



31 



EP0 901 105 A1 
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can be seen by the camera. Steps S5S4 to S598 are repeated until all the points have been processed in the manner 
described above. At step $600, CPU 4 determines whether there is another camera in the sequence. Steps S592 to 
S600 are repeated until all of cameras have been considered to remove surfaces as described above. 
[0305] In the processing described above, at step S594, CPU 4 projects the ray from a camera to the edge of the 

5 error ellipsoid for a point (rather than to the point itself) and considers whether the ray intersects any surface. This 
provides the advantage that the positional error for a point is taken into account. For example, if the ray was projected 
all the way to a point, a surface lying between the point and the edge of its error ellipsoid nearest to the camera would 
be intersected by the ray and hence removed. However this may produce an inaccurate result since the 30 point could 
actually tie anywhere in its error ellipsoid and could therefore be in front of the surface. The processing in the present 

J0 embodiment takes account of this. 

[0306] At step S602. CPU 4 considers the remaining triangular surfaces, and removes any which does not have a 
surface touching free space (this corresponding to a surface which is enclosed within the interior of the object). 
[0307] This is performed using a conventional technique, for example as described in 'Three-Dimensional Computer 
Vision" by Faugeras at Chapter 10, MIT Press. ISBN 0-262-06158-9. 

7 5 [0308] After performing steps S590 to S602, CPU 4 has produced a plurality of surfaces in a three-dim ensional space 
representing the object 24. At steps S604 to S6 10, CPU 4 determines the texture to be displayed on each triangular 
surface. 

[0309] At step S604, CPU 4 calculates the normal to the next remaining triangle (this being the first remaining triangle 
the first time step S604 is performed). At step S606, CPU 4 calculates the dot product between the normal calculated 

20 at step S604 and the optical axis of each camera to identify the camera which viewed the triangle closest to normal 
(this being the camera having the smallest angle between its optical axis and the normal to the surface). At step S608, 
CPU 4 reads the data for the camera identified in step S606 (previously stored at step $1 8 in Figure 4) and reads the 
image data lying between the vertices of the triangle to determine the texture for the triangle. At step S610, CPU 4 
determines whether there is another remaining triangle (or which the texture is to be determined. Steps S604 to S610 

& are repeated until the texture has been determined for aD triangles. 

[0310] Referring again to Figure 3, in this embodiment, after generating the surfaces representing the object at step 
S12, CPU 4 displays the surfaces at step S14. This is performed in a conventional manner, for example as described 
in 'Computer Graphics Principle and Practice* by Foley, van Dam, Feiner & Hughes, Second Edition, Addison- Wesley 
Publishing Company Inc., ISBN 0-201-12110-7. This process is summarised below. 

so [03 11] Figure 50 shows the opera I ions performed by CPU 4 is displaying the surface data at step Si 4. Referring to 
Figure 50, at step SS20, CPU 4 calculates the lighting parameters for the object, that is the data defining how the object 
is to be lit. This data may be input by a user using the input device 14, or, alternatively, default lighting parameters may 
be used. At step S622, the direction from which the object is to be viewed is defined by the user using input device 14. 
[031 2] At step S624, the vertices defining the planar triangular surfaces of the object are transformed from the object 

3S space in which they are defined into a modelling space in which the light sources are defined. At step S626, the 
triangular surfaces are lit by processing the data relating to the position of the light sources and the texture data for 
each triangular surface (previously determined at step S608). Thereafter, at step S628, the modelling space is trans- 
formed into a viewing space in dependence upon the viewing directed selected at step S622. This transformation 
identifies a particular field of view, which will usually cover less than the whole modelling space. Accordingly; at step 

40 S630, CPU 4 performs a clipping process to remove surfaces, or parts thereof, which fall outside the field of view 
[031 3] Up to this stage, the object data processed by the CPU 4 defines three-dimensional co-ordinate locations. At 
step S632, the vertices of the triangular surfaces are projected to define a two-dimensional image. 
[0314] After projecting the image into two dimensions, it is necessary to identify the triangular surfaces which are 
'front-facing*, that is facing the viewer, and those which are 'back-facing', that is cannot be seen by the viewer. There- 

45 fore, at step S634, back-facing surfaces are identified and culled. Thus, after step S634, vertices are defined in two 
dimensions identifying the triangular surfaces of visble polygons. 

[0315] At step S636, the two-dimensional data defining the surfaces is scan-converted by CPU 4 to produce pixel 

values, taking into account the data defining the texture of each surface previously determined at step S608 in Figure 49. 

[031 6] At step S638, the pixel values generated at step S636 are written to the Irame buffer on a surface-by-surface 
so basis, thereby generating data for a complete two-dimensional image. 

[0317] At step S640, CPU 4 generates a signal defining the pixel values. The signal is used to generate an image 

of the object on display unit 18 ancVor is recorded, for example on a video tape in video tape recorder 20. The signal 

may also be transmitted to a remote receiver for display or recording. 

[031 8] Various modifications are possible to the embodiment described so far. 
55 [031 9] In the embodiment above, as described with reference to Figure 2, camera 1 2 is moved to different positions 

about object 24 in order to record the images of the object. Instead, camera 12 may be maintained in a fixed position 

and object 24 moved relative thereto. Ol course, the positions of the camera 12 and the object 24 may both be moved 

to record the images. 
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. [0329] In the embodiment dJR2SSlir^* B ^ CPU - cateula ' ed Potato 
- tests the physical fundamenta, ma.r,7 ( sC 2^^? " anS '° mla,i0n * S,Bp S240 ' CPU 4 
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can be compared against a threshold to determine whether the pixel having that similarity measure matches the point 
of the first image. If a match is found, similar processing can be carried cut to determine whether a match can be found 
with a point in the third image, thereby identifying a triple of points. 

[0331] In the embodiments described above, when performing affine initial feature matching on a pair of images at 
5 step S62 or S64 in Figure 7, CPU 4 considers points in the first image of the pair which have bean matched with points 
in the preceding image in the sequence but which have not yet been matched with a point in the second image of the 
pair, and performs processing to try to match such points with points in the second image of the pair (steps SI 66 to 
S 176 in Figure 16). Thus, CPU 4 performs processing to "propagate" matched points through the sequence of images 
from a current image to a succeeding image in the sequence. It is also possible to perform such processing to 'prop- 

J0 agate' points in the opposite direction, that is, from a current image to a preceding image in the sequence. For example, 
. the images in the sequence could be considered in reverse order, that is, starting with the final image in sequence (the 
image taken at position L5 in the example of Figure 2), and the data processed in a similar manner to that already 
described. Processing can also be performed to "propagate" points in both directions, this being likely to provide more 
matches between points than when processing is performed to "propagate* points in a single direction. This, in turn, 

'5 may enable more accurate camera transformations to be calculated at step S66 in Figure 7. 

[0332] In the embodiment above, when CPU 4 performs constrained feature matching at step S74 in Figure 7. new 
matches between points in the second and third images of a triple of images may be identified at step S500 in Figure 
39. As explained previously, these points are considered in subsequent processing since the pair of images across 
which the new points are matched becomes the first pair of images in the next triple of images considered. Thus, when 

20 automatic initial feature matching or affine initial feature matching for the second pair of images in the next triple is 
performed at step S54 or step S64, the new matched points from the constrained feature matching may be used to 
identify matching points in the third image of the triple, as described above. On the other hand, in the embodiment 
above, the new matches generated at step $502 in Figure 39 between points in the first and second images of a triple 
when CPU 4 performs constrained feature matching are not considered in any subsequent initial feature matching 

25 operations. This is because the new matches are across the first pair of images in the triple, and this pair is not con- 
sidered further in subsequent initial leature matching processing. The new matches are. however, taken into account 
when CPU 4 generates the 3D data at step S10 (Figure 3) since the newly matched points form part of a "triple* points. 
As a modification, it is possible to perform additional processing to recalculate the camera transformations taking into 
account any new matches identified during constrained feature matching. This would produce two solutions for the 

30 camera transformations for each triple of images: the first being produced in the manner described above with respect 
to Figure 7, and the second being produced by the additional processing to take into account the new matches. The 
most accurate solution between the two may then be selected. 

[0333] In the embodiment described, in steps S52, S54, S60, S62, S64, S72 and S74 points (corner points, minimum 
points, maximum points, saddle points etc.) are matched in the images. However, it is possible to identify and match 
35 other 'features", for example lines etc. 

[0334] At step S528 in the embodiment above, CPU 4 merges points if they lie within one standard deviation of each 
other. However, it is possible to delete one of the points instead of combining them. 

[0335] In the embodiment described, having generated the surfaces at step S12 in Figure 3, CPU 4 performs process- 
ing to display the surface data at step 14. Alternatively, or in addition, instead of displaying the surface data at step 

40 S1 4, CPU 4 may: control manufacturing equipment to manufacture a model of the object 24, for example by controlling 
cutting apparatus to cut material to the a^ropriate dimensions; perform processing to recognise the object, for example 
by comparing it to data stored in a database; carry out processing to measure the object, for example by taking absolute 
measurements to record the size of the object, or by comparing the model with models of the object previously generated 
to determine changes therebetween; carry out processing so as to control a robot to navigate around the object; transmit 

45 the object data representing the model to a remote processing device for such processing (for example, CPU 4 may 
transmit the object data in VRML format over the Internet, enabling it to be processed by a WWW browser). Of course, 
the object data may be utilised in other ways. 

[0336] The techniques described above can be used in terrain mapping and surveying, with the three-dimensional 
data being input to a geographic information system (GIS) or other topographic database for example. 

so . 

Claim* 

1. In an image processing apparatus having a processor for processing first input signals defining an affine transfer- 
55 mation between a first pair of images of an object, second inpul signals defining a perspective transformation 

between the first pair of images, third input signals defining an affine transformation between a second pair of 
images of the object, one of the images being common to the first pair and the second pair, fourth input signals 
defining a perspective transformation between the second pair ol images, and fifth input signals defining features 
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12. A method according to any preceding claim, further comprising the step of processing image data defining the 
images cf the object to generate the input signals. 

13. A method according to any preceding claim, further comprising the step of generating object data defining a model 
s of the object in a three-dimensional space. 

14. A method according to claim 13, further comprising the step of processing the object data to generate image data. 

15. A method according to claim 14, further comprising the step of displaying an image of the object. 

16. A method according to claim 14 or claim 15, further comprising the step of recording the image data. 

17. A method according to any of claims 13 to 16, further comprising the step of transmitting a signal conveying the 
object data. 

J5 

18. A method according to any of claims 1 3 to 17, further comprising the step of recording the object data 

19. A method of operating an image processing apparatus to process signals defining first and second types of trans- 
formations between a first image and a second image and between a second image and a third image, and signals 

20 defining corresponding features in the images, so as to determine the relationship between all three images, the 

method comprising: 

determining the relationship using the corresponding features on the basis of the first type of transformation 
between the first and second images and the second type of transformation between the second and third 
& images; 

determining the relationship using the corresponding features on the basis of the first type of transformation 
between the first and second images and the first type of transformation between the second and third images; 
determining the relationship using the corresponding features on the basts of the second type of transformation 
between the first and second images and the second type of transformation between the second and third 
30 images; 

determining the relationship using the corresponding features on the basis of the second type of transformation 
between the first and second images and the first type of transformation between the second and third images; 
and 

selecting the most accurate relationship. 

35 

20. A method of operating an image processing apparatus to process signals defining (i) first and second types of 
transformation between a first image and a second image, (ii) a transformation between the second image and a 
third image, and (iii) corresponding features in the images, so as to determine the relationship between all three 
images, the method comprising: 

40 

determining the relationship using the corresponding features on the basis of the first type of transformation 
between the first and second images and the transformation between the second and third images; 
determining the relationship using the corresponding features on the basis of the second type of transformation 
between the first and second images and the transformation between the second and third images; and 
4S selecting the most accurate relationship. 

21. In an image processing apparatus having a processor for processing first input signals defining transformations 
between at least three images of an ojbect arranged in pairs with each pair of images containing an image which 
is part of another pair, the first input signals defining (i) a respective transformation of a first type between the 

so images in each of the pairs and (ii) a respective transformation of a second type between the images in each of 

the pairs, and second input signals defining features matched in images, a method of processing the input signals 
to produce signals defining a transformation between all of the images, the method comprising: 

(a) calculating for each respective combination of the transformations between the images defined in the first 
ss input signals a transformation between ail images using matching features defined in the second input signals; 

and 

(b) selecting the most accurate calculated transformation. 
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(a) means for calculating the transformation between all three images using the first, second and fourth input 
signals; 

(b) means for calculating the transformation between atl three Images using the first, third and fourth input 
signals; and 

s (c) means for selecting the most accurate calculated transformation. 

28. Apparatus according to claim 26 or claim 27, wherein a transformation between all three images is calculated by 
calculating the transformation scale for at least one value of the rotation angle between the "mags 6 in the first pair 
and the rotation angle between the images in the second pair. 

w 

29. Apparatus according to claim 28, wherein a plurality of transformation scales are calculated for each value of the 
rotation angles, wherein the accuracy of each calculated scale is determined, and wherein the scale with the 
highest determined accuracy is selected. 

Apparatus according to claim 28 or claim 29, wherein, when the transformation between all three images is cal- 
culated using signals defining perspective -affme transformations between the images, affine-perspective transfor- 
mations between the images or afflne-affin© transformations between the images, the transformation scale is cal- 
culated for a plurality of values of the rotation angles, the accuracy of each calculated scale is determined, and 
the scale and value of the rotation angles with the highest accuracy is selected. 

Apparatus according to claim 30, wherein the plurality of values of the rotation angles comprise values of at least 
one of the rotation angle between the images in the first pair and the rotation angle between the images in the 
second pair distributed over a range. 

& 32. Apparatus according to any of claims 28 to 31 , wherein a transformation scale is calculated by calculating the 
transformations and associated scale which causes a ray projected from a feature in a first of the three images, a 
ray projected from the matched feature in a second of the images, and a ray projected from the matched feature 
in a third of the images to intersect at a point. 

JO 33. Apparatus according to any of claims 28 to 32, wherein the accuracy of a calculated transformation scale is de- 
termined by calculating the distance between (i) the position of a calculated feature in a first of the images deter- 
mined in dependence upon matched features in the second and third images and (ii) the position of the matched 
feature in the first image. 

. 3S 34. Apparatus according to claim 33, wherein the position of the calculated feature in the first image is determined by 
projecting a ray from a feature in the second image and a ray from the matched feature in the third image, calculating 
a point in three-dimensional space in dependence upon the projected rays, and calculating the position of the 
feature in the first image in dependence upon the point in the three-dimensional space. 

•to 35. Apparatus according to any of claims 26 to 34, wherein the features defined in the fifth input signals comprise points. 

36. Apparatus according to any of claims 26 lo 35, further comprising means for converting the selected most accurate 
calculated transformation into a rotation matrix and translation vector. 



& 37. Apparatus according to any of claims 26 to 36, further comprising means for processing image data defining images 
of the object to generate the input signals. 

3B. Apparatus according to any of claims 26 to 37, further comprising means for generating object data defining a 
model of the object in a three-dimensional space. 

so 

39. Apparatus according to claim 38, further comprising means for processing the object data to generate image data. 

40. Apparatus according to claim 39, further comprising means for displaying an image of the object 

55 41. An image processing apparatus for processing first input signals defining transformations between at least three 
images of an ojbect arranged in pairs with each pair of images containing an image which is part of another pair; 
the first input signals defining (i) a respective transformation of a first type between the images in each of the pairs 
and (ii) a respective transformation of a second type between the images in each of the pairs, and second input 
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(c) selecting the most accurate calculated relationship. 

49. A method according to claim 48, wherein, in each of steps (a) and (b), the perspective relationship between the 
camera positions is calculated. 

5 

50. A method according to claim 48, wherein, in each of steps (a) and (b), the affine relationship between the camera 
positions is calculated. 

51. A method according to any of claims 48 to 50, wherein each of steps (a) and (b) lurther comprises testing the 
io calculated relationship using pairs of features matched by the first matching technique and pairs of features 

matched by the second matching technique. 

52. A method according to any of claims 48 to 51, wherein, in each of steps (a) and (b). the pairs of features used lor 
calculating the camera positions are selected at random or in a pseudo-random way. 

15 

53. A method according to any of claims 48 to 52, wherein the pairs of features matched by one of the matching 
techniques have been matched by a user, and the pairs ot features matched by the other malching technique have 
been matched by an image processing apparatus. 

20 54. A method according to any of claims 48 to 53, further comprising the step of processing image data defining the 
images of the object to generate the input signals. 

55. A method according to any of claims 48 to 54, wherein the pairs of features comprise pairs of points. 

& 56. A method according to any of claims 48 to 55, further comprising the step of processing the iiput signals and the 
signals defining the relationship between the camera positions to generate object data defining a model of the 
object in a three-dimensional space. 

57. A method according to claim 56, further comprising the step of processing the object data to generate image data. 

30 

58. A method according to claim 57, further comprising the step of displaying an image of the object. 

59. A method according to claim 57 or claim 58, further comprising the step of recording the image data. 

3£ 60. A method according to any of claims 56 to 59, further comprising the step ol transmitting a signal conveying the 
object data. 

61. A method according to any of claims 56 to 60, further comprising the step of recording the object data. 

40 62. A method of operating an image processing apparatus to process first signals defining object features matched 
with a first matching technique in first and second images taken from imaging positions of undefined relationship 
and second signals defining object features matched in the first and second images with a second matching tech- 
nique, so as to determine the positional relationship between the images, the method comprising: 

45 (a) processing the first input signals to determine the relationship between the images; 

(b) processing the second input signals to determine the relationship between the images; 

(c) selecting the most accurate determined relationship. 

63. A method according to claim 62. wherein, in step (b), the first input signals and the second input signals are 
so processed to determine the relationship between the images. 

64. An image processing apparatus for processing input signals defining a plurality of pairs of features representing 
features matched in first and second images of an object taken from undefined camera positions using first and 
second matching techniques to produce signals defining the relationship between the camera positions, compris- 
es ing: 

(a) means arranged to use pairs of features matched by the first matching technique to calculate, at least to 
some extent, the relationship between the camera positions; 
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if the calculated physically realisable matrix is consistent with a predetermined number of the pairs of matched 
features used to calculate the fundamental matrix, then the physically realisable matrix is tested against other 
pairs of matched features defined in the input signals. 

80. A method according to claim 79, wherein the predetermined number comprises all of the pairs of matched features 
used to calculate the fundamental matrix. 

61. A method according to any of claims 77 to 80, wherein, in step (c), the calculated physically realisable matrix is 
tested using each pair of matched features defined in the input signals. 

82. A method according to any of claims 77 to 81 , wherein, in step (d), steps (a) to (c) are repeated a plurality of times, 
the number of repetitions being determined in accordance with the number of pairs of leatures defined in the input 
signals. 

83. A method according to claim 82. wherein the number of times steps (a) to (c) are repeated is a percentage of the 
maximum number of different combinations ol the number of pairs of features used in step (a) to calculate the 
fundamental matrix it is possible to select from the pairs of features defined in the input signals. 

84. A method according to claim 82 or claim 83, wherein, in step (d), the repetition of steps (a) to (c) is stopped if it is 
determined in step (c) thai the accuracy of the calculated physically realisable matrix has not increased in a given 
number of previous iterations. 

85. A method according to any of claims 77 to 84, wherein the input signals define at least eight pairs of matching 
features identified using a first matching technique and at least one pair ol matching features identified using a 
second matching technique, and wherein the method comprises: 

(i) performing steps (a) to (e) selecting the pairs of features to be used in step (a) from those identified with 
the first matching technique; 

(ii) performing steps (a) to (e) selecting the pairs of features to be used in step (a) from those identified with 
the first matching technique and those identified with the second matching technique; and 

(iii) selecting the most accurate calculated physically realisable matrix from the physically realisable matrix 
selected in step (i)(e) and the physically realisable matrix selected in step (ii)(e). 

66. A method according to any of claims 77 to 84, wherein the input signals define at least eigfrt pairs of matching 
features identified using a first matching technique and at least eight pairs of matching features identified using a 
second matching technique, and wherein the method comprises: 

(0 performing steps (a) to (e) selecting pairs of features to be used in step (a) from those identified with the 
first matching technique; 

(ii) performing steps (a) to (e) selecting pairs of features to be used in step (a) from those identified with the 
second matching technique; and 

(iii) selecting the most accurate calculated physically realisable matrix from the physically realisable matrix 
selected in step (i)(e) and the physically realisable matrix selected in step (ii)(e). 

67. A method according to claim 85 or claim B6, wherein the pairs of features matched by one of the matching tech* 
niques have been matched by a user and the pairs of features matched by the other matching technique have 
been matched by an image processing apparatus. 

86. A method according to any of claims 77 to 87, further comprising the step of converting the selected most accurate 
physically realisable matrix into a rotation matrix and translation vector. 

89, A method according to any of claims 77 to 88, further comprising the step of processing image data defining the 
images of the object to generate the input signals. 

9a A method according to any of claims 77 to 89, wherein the physically realisable matrix comprises the physical 
fundamental matrix. 

91. A method according to any of claims 77 to 90, wherein pairs of features comprise pairs of points. 
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106.Apparatus according to claim 104, wherein the number of times the operations performed by means (a) to (c) are 
repeated is a percentage of the maximum number of different combinations of the number of pairs of features used 
by means (a) to calculate the fundamental matrix it is possible to select from the pairs of lealures defined in the 
input signals. 

5 

106. Apparatus according to claim 104 or claim 105, controlled such that the repetition of the operations by means (a) 
to (c) is stopped if means (c) determines that the accuracy of the calculated physically realisable matrix has not 
increased in a given number of previous iterations. 

'0 107. Apparatus according to any of claims 99 to 106, wherein the input signals define at least eight pairs of matching 
features identified using a first matching technique and at least one pair of matching features identified ushg a 
second matching technique, and wherein the apparatus is controlled such that: 

(0 means (a) to (d) are operated selecting the pairs of features to be used by means (a) from those identified 
with the first matching technique to give a first selected physically realisable matrix; 

(ii) means (a) to (d) are operated selecting the pairs of features to be used by means (a) from those identified 
with the first matching technique and those identified with the second matching technique to give a second 
selected physically realisable matrix; and 

(iii) the most accurate calculated physically realisable matrix from the first and second selected physically 
20 realisable matrices is selected. 

108. Apparatus according to any of claims 99 to 106, wherein the input signals define at least eight pairs of matching 
features identified using a first matching technique and at least eight pairs of matching features identified using a 
second matching technique, and wherein the apparatus is controlled such that: 

2$ 

(i) means (a) to (d) are operated selecting pairs of features to be used by means (a) from those identified with 
the first matching technique to give a first selected physically realisable matrix: 

(ii) means (a) to (d) are operated selecting pairs of features to be used by means (a) from those identified with 
the second matching technique to give a second selected physically realisable matrix; and 

30 (iii) the most accurate calculated physically realisable matrix from the first and second selected physically 

realisable matrices is selected. 

109. Apparatus according to claim 107 or claim 108, wherein the pairs of features matched by one of the matching 
techniques have been matched by a user, and the pairs of features matched by the other matching technique have 

-3-5 been matched by an image processing apparatus. 

11 0. Apparatus according to any of claims 99 to 109, further comprising means for converting the selected most accurate 
physically realisable matrix into a rotation matrix and translation vector. 

40 111 .Apparatus according to any of claims 99 to 110, further comprising means for processing image data defining the 
images of the object to generate the input signals. 

112. Apparatus according to any of claims 99 to 111. wherein the physically realisable matrbc comprises the physical 
fundamental matrix. 

45 

11 3. Apparatus according to any of claims 99 to 112, wherein pairs of features comprise pairs of points. 

11 4. Apparatus according to any of claims 99 to 1 13, further comprising means for processing the input signals and the 
signals defining the relationship between the camera positions to generate object data defining a model of the 

50 object in a three-dimensional space. 

11 5. Apparatus according to claim 114, further comprising means for processing the object data to generate image data. 
11a Apparatus according to claim 115, further comprising means for displaying an image of the object. 

55 

11 7. A storage device storing instructions for causing a programmable processing apparatus to perform a method ac- 
cording to any of claims 77 to 98. 
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127. A method according to claim 126, wherein the probability distribution is a Gaussian distribution. 

128. A method according to claim 126 or claim 127. wherein step (iii) comprises calculating each single point in the 
three-dimensional space by combining the probability distribution of a re-pcsitioned point and the probability dis- 

5 iributlon, If any. of the point In the three-dimensional space generated in the other of steps (a) and (b) from the 

corresponding matched image points. 

129. A method according to any of claims 1 26 to 123, wherein, in step (d), it is determined that a first of the single points 
represents the same point on the object as a second of the single points if the first point lies within a given distance 

io of the second point, the given distance being dependent upon the positional error probability distribution of the 

second point. 

130. A method according to claim 129, wherein the given distance is the Mahalanobis distance of the probability dis- 
tribution of the second point. 

15 

1 31 .A method according to any of claims 1 1 9 to 1 30. wherein, in step (e), the one point in the three-dimensional space 
is calculated in dependence upon the positions of all the single points which may represent the same point on the 
object and their associated positional errors. 

20 132.A method according to any of claims 119 to 131, further comprising the step of processing image data defining 
the images of the object to generate the first input signals. 

133. A method according to any of claims 11 9 to 1 32. further comprising the step of processing the first input signals 
to generate the second input signals. 

25 

134. A method according to any of claims 1 1 9 to 1 33, further comprising the step of processing the signals defining the 
points in the three-dimensional space representing points on the object to generate image data. 

135. A method according to claim 1 34, further comprising the step of displaying an image of the object. 

30 

136. A method according to claim 1 34 or claim 1 35, further comprising the step of recording the image data. • 

137. A method according to any of claims 119 to 1 36. further comprising the step of transmitting the signals defining 
the points in the three-dimensional space representing points on the object. 

35 

138. A method according to any of claims 119 to 137, further comprising the step of recording the signals defining the 
points in the three-dimensional space representing points on the object 

139. A method of operating an image processing apparatus to process first input signals defining a first plurality of 
40 points comprising a point in each of first, second and third images ol an object, second input signals defining a 

second plurality of points comprisir further point in each of the first, second and third images, and third input 
signals defining the relationship between the positions at which the first, second and third images were recorded, 
so as to define points in a three-dimensional space representing points on the object, the method comprising: 

-*£ processing the first and third input signals to define a first point in the three-dimensional space on the basis 

of the points in the firs! and second images and a second point in the three-dimensional space on the basis 
of the points in the second and third images; 

processing the second and third input signals to define a third point in (he three-dimensional space on the 
basis of the further points in the first and second images and a fourth point in the three-dimensional space on 
50 - the basis of the further points in the second and third images; 

defining a fifth point in the three-dimensional space in dependence upon the first and second points in the 
three-dimensional space; 

defining a sixth point in the three-dimensional space in dependence upon the third and fourth points in the 
three-dimensional space; 

S5 determining whether the fifth and sixth points in the three-dimensional space may represent the same point 

on the object, and, if so, defining a seventh point in the three-dimensional space in dependence upon the fifth 
and sixth points. 
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dimensional space using .he po*. in Z^aeTZ ^ °" 3 p0in ' in th « 

(c) means lor calculating a singTe Bo"n.T.h?fhll^ f ^ " ^ 0ther ima 9 9 <* ,he 
poin.ma.ched in each of V" IT r!^ 10 " 3 '^"^ associated positional error for each 

by™ans(a)andmepoin.S 

K image' ^ ^ 3 <* - "« « — - - came, .or 

calculating ft. mid-poin. of .he line which is perpendicular to bo.h .he projeCed ,ay S . 
W m e r e ^Xac: b r " C ' am 141 ' ^ meanS (C> " affanged * «*- - ** in 
ITmZT*™ Brr0 ' '°' P ° W " ^ -P- by - . ea , one o. 

^CS,r h ^r^ ^ a re.pos.oned poi, 

ing matched image points, aerated by the other o( means (a) and (b) from the correspond- 

143.Appara.us according to claim 142. wherein step (i) comprises' 

146.Appara.us according to any of claims 142 to 1 45 whamin • ■ 

associated with saw single points M^on!£S^i ( iT anBnBed '° Ca,CUlate 1,16 P 08 **" 3 ' 9 "°r. 

148.Appara.us according to Cairn 147, wherein tne probabilfty distribution is a Gausssn distribution. 



(a) 



with 
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149. Apparatus according to claim 147 or claim 143, wherein step (iii) comprises calculating each single point in the 
three-dimensional space by combining the probability distribution cf a re-pcsitioned point and the probability dis- 
tribution, if any, of the point in the three-dimensional space generated by the other ot means (a} and (b) I ram the 
corresponding matched image points. 

150. Apparatus according to any of claims 147 to K9, wherein means (d) is arranged to determine that a first of the 
single points represents the same point on the object as a second of the single points if the first point lies within 
a given distance of the second point, the given distance being dependent upon the positional error probability 
distribution of the second point 

151. Apparatus according to claim 150, wherein the given distance is the Mahalanobis distance of the probability dis- 
tribution of the second point. 

152. Apparatus according to any of claims 140 to 151. wherein means (e) is arranged to calculate the one point in the 
three-dimensional space in dependence upon the positions of all the single points which may represent the same 
point on the object and their associated positional errors. 

153. Apparatus according to any of claims 140 to 152, further comprising means for processing image data defining 
images ot the object to generate the first input signals. 

154. Apparatus according to any of claims 140 lo 153, further comprising means for processing the first input signals 
to generate the second input signals. 

155. Apparalus according to any of claims 140 to 1 54, further comprising means for processing the signals defining 
2S the points in the three-dimensional space representing points on the object to generate image data. 

156. Apparatus according to claim 155, further comprising means for displaying an image of the object. 

157. A storage device storing instructions for causing a programmable processing apparatus to perform a method 
30 according to any of claims 1 1 9 to 1 39. 

158. A signal for causing a programmable processing apparatus to perform a method according to any of claims 119 
to 139. 

35 
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READ MATCHED POINTS FOR SECOND PAIR 
OF IMAGES 



I 





GENERATE LISTS OF CORRESPONDING 
POINTS AS FOLLOWS FOR SECOND PAIR OF 
IMAGES: 

(i) USER - IDENTIFIED POINTS 

(ii) USER - IDENTIFIED & CALCULATED 
POINTS 

GENERATE UST OF TRIPLE" POINTS 
(MATCHED IN ALL THREE IMAGES) 


S464 






r 






NORMALISE POINTS 


S466 






r 






SET UP MEASUREMENT MATRIX FOR EACH 
LIST OF POINTS 


S468 








r 






DETERMINE NUMBER OF ITERATIONS TO 86 PERFORMED FOR 
THE FOLLOWING CALCULATIONS FOR THE SECOND PAIR OF 
IMAGES: 






(i) PERSPECTIVE CALCULATION FOR USER-IDENTIFIED POINTS 

(ii) PERSPECTIVE CALCULATION FOR USER-IDENTIFIED 4 
CALCULATED POINTS 

(iti) AFFINE CALCULATION FOR USER-IDENTIFIEO POINTS 
(iv) AFFINE CALCULATION FOR USER-IDENTIFIED & 
CALCULATED POINTS 





S470 



Fig. 37. 



83 



EP0 901 105 A1 



CONStOER ♦ - AFFINE CASE AND CALCULATE S & p2 S472 



S474 



S476 



CONSIDER ♦ - PERSPECTIVE CASE AND CALCULATE S 



SELECT MOST ACCURATE SOLUTION 



YES 



S480 



CAMERA 
TRANSFORMATIONS 
ARE NOT 
SUFFICIENTLY 
ACCURATE 



_ S478 

IS NUMBER 
OF POINTS WHICH ARE CONSISTENT WITH 
MOST ACCURATE SOLUTION < 4? 



_ S482 

IS NUMBER 
OF POINTS WHICH ARE CONSISTENT WITH 
MOST ACCURATE SOLUTION > 80% OF 
ALL TRIPLE POINTS? 



NO 



YES 



IS MOST 

ACCURATE SOLUTION ♦ - PERSPECTIVE 
CASE? 

NO 



S484 



YES 



OPTIMISE SOLUTION 



S486 



CALCULATED CAMERA TRANSFORMATIONS ARE 
SUFFICIENTLY ACCURATE . CONVERT TO CAMERA ROTATION 
AND TRANSLATION 



S488 



i r 

Fig. 38. 
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k m 

FOR "DOUBLE" POINTS IN FIRST AND SECOND 

IMAGES OF CURRENT TRIPLE OF IMAGES, TRY Ci;nn 

TO IDENTIFY A CORRESPONDING POINT IN b:>U0 
THIRD IMAGE 

» 

FOR -DOUBLE" POINTS IN SECOND AND THIRD 
IMAGES OF CURRENT TRIPLE OF IMAGES, TRY 
TO IDENTIFY A CORRESPONDING POINT IN 5)51,2 
FIRST IMAGE 

Fig. 39. 



PROJECT NEXT POINT IN SECOND IMAGE WHICH 
FORMS A "DOUBLE" POINT WITH THE OTHER IMAGE OF 
THE PAIR INTO THE REMAINING IMAGE OF THE TRIPLE 
USING THE CALCULATED CAMERA TRANSFORMATIONS 



CALCULATE SIMILARITY MEASURE BETWEEN POINTS 
LYING WITHIN + SET ERROR IN X DIRECTION AND ± SET 
ERROR IN Y DIRECTION OF PROJECTED POINT AND THE 
POINT IN THE SECOND IMAGE 



S504 



S506 



NO 



IS HIGHEST 
SIMILARITY MEASURE GREATER 
THAN THRESHOLD? 



S510 



Yes 



FORM TRIPLE 



□ 



S512 



S514 



YES 



ANOTHER 
DOUBLE POINT? 



Fig. 40. 
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FOR EACH PAIR OF IMAGES, CALCULATE 3D PROJECTION 
OF EACH USER-IDENTIFIED DOUBLE OR POIfNfTS WHICH 
FORM PART OF A TRIPLE WIT* A SUBSEQUENT IMAGE 



IDENTIFY AND DISCARD INACCURATE 3D POINTS AND 
CALCULA 1 E ERROR FOR EACH PAIR OF CAMERA 
POSITIONS 



ADJUST EACH 3D POINI FOR CAMERA POSITION ERROR 



i £2^.L*ll T J2 )MB,NEO 30 P0,NTS CORRESPOND TO 
UNIQUE IMAGE POINTS AND MERGE ONES THAT DO NOT 



S520 



S522 



S524 



COMBINE 3D POINTS WHICH ARE FROM A COMMON IMAGE 
POINT S526 



S528 



Fig. 41 
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CONSIDER NEXT PAIR OF IMAGES 



S530 



PROJECT 30 LINE FROM EACH POINT IN NEXT PAIR OF POINTS IN 

CURRENT PAIR OF IMAGES WHICH FORM A USER-IDENTIFIED 
DOUBLE OR PART OF A TRIPLE OF POINTS WITH A SUBSEQUEf^T 
IMAGE 



S532 



CALCULATE MID-POINT OF LINE WHICH CONNECTS. AND IS 
PERPENDICULAR TO. DOTH PROJECTED LINES 



S534 



HAS A —^_S536 
CORRESPONDING POINT BEEN MATCHED IN 
NEXT IMAGE? 



YES 



PROJECT 3D LINE FROM MATCHED POINT IN NEXT IMAGE 

+ 

CALCULATE MID-POINT OF LINE WHICH CONNECTS. AND IS 
PERPENDICULAR TO, THE NEW PROJECTED UNE AND THE 
PROJECTED LINE FROM THE PREVIOUS IMAGE 



S538 



S540 



YES 



HAS A _ 
CORRESPONDING POINT BEEN MATCHED IN 
NEXT IMAGE? 



S542 



NO 



YES 



ANOTHER PAIR OF 5544 
DINTS NOT PREVIOUSLY CONSIDERE& 
IN CURRENT PAIR OF IMAGES WHICH FORM 
JK USER-IDENTIFIED DOUBLE OR PART OF A TRIPLE 
^ OF POINTS WITH A SUBSEQUENT 

IMAGE? 



YES 



S546 



ANOTHER PAIR OF IMAGES? 



Fig. 42. 
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CONSIDER ALL 30 POINTS. AND CALCULATE THE STANDARD DEVIATION 
OF THE X. y and 2 COORDINATES -> AX. Ay. A2 



S550 



CALCULATE OBJECT SIZE = (AX 2 + Ay 2 * AZ 2 ) 1/2 



S552 



FOR NEXT PAIR OF CAMERA POSITIONS, CONSIDER NEXT 30 POINT 
ORIGINATING FROM A TRIPLE OF POINTS WITH A SUBSEQUENT IMAGE 
AND CALCULATE SHIFT BETWEEN THIS 3D POINT AND CORRESPONDING 
POINT PREVIOUSLY CALCULATED FOR SUBSEQUENT PAIR OF CAMERA 

POSITIONS 



S554 



S558 



IS MAGNITUDE 
OF SHIFT > 10% OF OBJECT SIZE? 



NO 



YES 



S560 



DISCARD BOTH 3D POINTS 



YES 



ANOTHER 

30 POINT FOR CURRENT PAIR OF CAMERA" 
^POSITIONS ORIGINATING FROM A TRIPLE 
OF POINTS? 



S562 



CALCULATE NET OF SHIFTS BETWEEN POINTS FOR CURRENT PAIR OF 
CAMERA POSITIONS AND POINTS FOR SUBSEQUENT PAIR OF CAMERA 

POSITIONS TO GIVE ERROR ROTATION MATRIX AND ERROR 
TRANSLATION VECTOR FOR SUBSEQUENT PAIR OF CAMERA POSITIONS 



ADJUST POINTS FOR SUBSEQUENT PAIR OF CAMERA POSITIONS USING 
CALCULATED ERROR TO GIVE CORRECTED 3D POINTS 



± 



CALCULATE DIFFERENCE BETWEEN EACH CORRECTED 3D POINT AND 

ITS CORRESPONDING POINT FOR CURRENT PAIR OF CAMERA 
POSITIONS, AND CALCULATE COVARIANCE MATRIX (ERROR ELLIPSOID) 
OF THE DIFFERENCES 



® 



S564 



S566 



S568 



Fig. 44. 
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Fig.45b. 
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Fig.46. 
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X 

#1 



#1 
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#1 



Fig.47. 
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SORT 3D POINTS IN ORDER OF SIZE OF ERROR ELLIPSOID 
(SMALLEST FIRST) 



S580 



COMPARE NEXT HIGHEST POINT IN LIST WITH ALL 
SUBSEQUENT POINTS AND IDENTIFY ALL SUBSEQUENT 
POINTS FOR WHICH HIGHEST POINT UNDER CONSIDERATION 
IS WITHIN A DISTANCE OF 1 x ITS MAHALANOBIS DISTANCE 



COMBINE HIGHEST POINT UNDER CONSIDERATION WITH 
EVERY IDENTIFIED POINT TO PRODUCE ONE COMBINED 
POINT. REPLACE HIGHEST POINT UNDER CONSIDERATION 
WITH COMBINED POINT. AND DISCARD IDENTIFIED POINTS 
USED TO CREATE COMBINED POINT 



S584 



YES 



ANOTHER 
POINT IN LIST NOT YET 
CONSIDERED? 



S586 



Fig. 48. 
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PERFORM OELAUNAY TRIANGULATION OF 30 POINTS 



CONSIDER NEXT CAMERA 



PROJECT RAY FROM CAMERA TO NEXT 30 POINT WHICH CAN 
I BE SEEN BY THAT CAMERA 



YES 



REMOVE ANY SURFACE THE RAY INTERSECTS 



S598 



ANOTHER 
POINT WHICH CAMERA CAN SEE? 



S590 

S592 

S594 
S596 



YES 



S600 



ANOTHER CAMERA? 



- NO 





REMOVE ALL TRIANGLES WHICH DO NOT HAVE A SURFACE 
TOUCHING FREE SPACE 


| S602 








~~ 


r 




| CALCULATE NORMAL TO NEXT REMAINING TRIANGLE 


S604 




■ ^ 






OALvUL* ituuf PRODUCT BETWEEN NORMAL AND OPTICAL 
AXIS OF EACH CAMERA AND IDENTIFY CAMERA WHICH 
VIEWED THE TRIANGLE CLOSEST TO NORMAL 


$606 










READ TEXTURE FOR TRIANGLE FROM DATA FOR IDENTIFIED* I 
CAMERA 


S608 


YFS^ 







ANOTHER TRIANGLE? 



Fig. 49. 



94 



EP0 901 105 A1 



1 



CALCULATE LIGHTING PARAMETERS 



DEFINE VIEWING DIRECTION 



± 



PERFORM LOCAL TRANSFORMATION 



LIGHT SURFACES 



PERFORM VIEW TRANSFORMATION 



CUP 

HI 



PROJECT TO DEFINE IMAGE IN 2-D 



CULL BACKFACES 



SCAN CONVERT TO PIXELS 



WRITE TO FRAME BUFFER 



2-0 VIDEO IMAGE 



S620 
S622 
S624 
S626 
S628 
S630 
S632 
S634 
S636 
S638 
S640 



Fig. 50. 
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